2024-03-25T16:14:15.240 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T16:14:15.240 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --owner scheduled_amathuri@teuthology --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621259 --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621259/orig.config.yaml 2024-03-25T16:14:15.284 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T16:14:15.369 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621259 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621259' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi156.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCkGg4kPQJMrq/462q94BLEniTaliNpRqetozH5/BjX8Z+YsVbRb9yqGlYNox0HbEpukRiW9mhXvMGQTIUutda4= smithi194.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL8bN/RvARiv4DZo2Ft70wIhEmZyDAQhciYVLOvdDAaTOzX9xpc3r1GJHnSvl0cmcRUaESnAh3PhVN2JQjiPfsE= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T16:14:15.369 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T16:14:15.371 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T16:14:15.372 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T16:14:15.373 INFO:teuthology.task.internal:Checking packages... 2024-03-25T16:14:15.431 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T16:14:15.431 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T16:14:15.431 INFO:teuthology.packaging:ref: None 2024-03-25T16:14:15.431 INFO:teuthology.packaging:tag: None 2024-03-25T16:14:15.431 INFO:teuthology.packaging:branch: main 2024-03-25T16:14:15.431 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:14:15.431 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T16:14:15.663 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T16:14:15.664 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T16:14:15.671 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T16:14:15.671 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T16:14:15.680 INFO:teuthology.task.internal:Saving configuration 2024-03-25T16:14:15.697 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T16:14:15.703 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T16:14:15.731 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621259', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:08:58.009581', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCkGg4kPQJMrq/462q94BLEniTaliNpRqetozH5/BjX8Z+YsVbRb9yqGlYNox0HbEpukRiW9mhXvMGQTIUutda4='} 2024-03-25T16:14:15.752 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621259', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:08:58.012069', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL8bN/RvARiv4DZo2Ft70wIhEmZyDAQhciYVLOvdDAaTOzX9xpc3r1GJHnSvl0cmcRUaESnAh3PhVN2JQjiPfsE='} 2024-03-25T16:14:15.752 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T16:14:15.759 INFO:teuthology.task.internal:roles: ubuntu@smithi156.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T16:14:15.759 INFO:teuthology.task.internal:roles: ubuntu@smithi194.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T16:14:15.759 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T16:14:15.828 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe7c67650d0>, signals=[15]) 2024-03-25T16:14:15.828 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T16:14:15.834 INFO:teuthology.task.internal:Opening connections... 2024-03-25T16:14:15.834 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi156.front.sepia.ceph.com 2024-03-25T16:14:15.836 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:14:15.933 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi194.front.sepia.ceph.com 2024-03-25T16:14:15.934 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:14:16.022 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T16:14:16.029 DEBUG:teuthology.orchestra.run.smithi156:> uname -m 2024-03-25T16:14:16.047 INFO:teuthology.orchestra.run.smithi156.stdout:x86_64 2024-03-25T16:14:16.047 DEBUG:teuthology.orchestra.run.smithi156:> cat /etc/os-release 2024-03-25T16:14:16.105 INFO:teuthology.orchestra.run.smithi156.stdout:NAME="CentOS Stream" 2024-03-25T16:14:16.105 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION="9" 2024-03-25T16:14:16.105 INFO:teuthology.orchestra.run.smithi156.stdout:ID="centos" 2024-03-25T16:14:16.105 INFO:teuthology.orchestra.run.smithi156.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:14:16.105 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION_ID="9" 2024-03-25T16:14:16.105 INFO:teuthology.orchestra.run.smithi156.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:14:16.105 INFO:teuthology.orchestra.run.smithi156.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:14:16.105 INFO:teuthology.orchestra.run.smithi156.stdout:ANSI_COLOR="0;31" 2024-03-25T16:14:16.106 INFO:teuthology.orchestra.run.smithi156.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:14:16.106 INFO:teuthology.orchestra.run.smithi156.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:14:16.106 INFO:teuthology.orchestra.run.smithi156.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:14:16.106 INFO:teuthology.orchestra.run.smithi156.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:14:16.106 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:14:16.106 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:14:16.107 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server 2024-03-25T16:14:16.126 DEBUG:teuthology.orchestra.run.smithi194:> uname -m 2024-03-25T16:14:16.143 INFO:teuthology.orchestra.run.smithi194.stdout:x86_64 2024-03-25T16:14:16.144 DEBUG:teuthology.orchestra.run.smithi194:> cat /etc/os-release 2024-03-25T16:14:16.200 INFO:teuthology.orchestra.run.smithi194.stdout:NAME="CentOS Stream" 2024-03-25T16:14:16.200 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION="9" 2024-03-25T16:14:16.200 INFO:teuthology.orchestra.run.smithi194.stdout:ID="centos" 2024-03-25T16:14:16.200 INFO:teuthology.orchestra.run.smithi194.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:14:16.201 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION_ID="9" 2024-03-25T16:14:16.201 INFO:teuthology.orchestra.run.smithi194.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:14:16.201 INFO:teuthology.orchestra.run.smithi194.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:14:16.201 INFO:teuthology.orchestra.run.smithi194.stdout:ANSI_COLOR="0;31" 2024-03-25T16:14:16.201 INFO:teuthology.orchestra.run.smithi194.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:14:16.201 INFO:teuthology.orchestra.run.smithi194.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:14:16.201 INFO:teuthology.orchestra.run.smithi194.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:14:16.201 INFO:teuthology.orchestra.run.smithi194.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:14:16.201 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:14:16.201 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:14:16.201 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server 2024-03-25T16:14:16.225 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T16:14:16.234 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T16:14:16.240 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T16:14:16.240 DEBUG:teuthology.orchestra.run.smithi156:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:14:16.243 DEBUG:teuthology.orchestra.run.smithi194:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:14:16.259 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T16:14:16.266 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T16:14:16.266 DEBUG:teuthology.orchestra.run.smithi156:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:14:16.301 DEBUG:teuthology.orchestra.run.smithi194:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:14:16.330 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T16:14:16.386 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T16:14:16.403 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T16:14:16.403 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T16:14:16.403 DEBUG:teuthology.orchestra.run.smithi156:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:14:16.404 DEBUG:teuthology.orchestra.run.smithi194:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:14:16.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:14:16.420 DEBUG:teuthology.orchestra.run.smithi156:> uname -r 2024-03-25T16:14:16.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:14:16.421 DEBUG:teuthology.orchestra.run.smithi194:> uname -r 2024-03-25T16:14:16.477 INFO:teuthology.orchestra.run.smithi156.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:14:16.477 INFO:teuthology.task.kernel:Running kernel on smithi156: 5.14.0-361.el9.x86_64 2024-03-25T16:14:16.477 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum install -y kernel 2024-03-25T16:14:16.479 INFO:teuthology.orchestra.run.smithi194.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:14:16.479 INFO:teuthology.task.kernel:Running kernel on smithi194: 5.14.0-361.el9.x86_64 2024-03-25T16:14:16.479 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum install -y kernel 2024-03-25T16:14:18.679 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-25T16:14:18.820 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS-9 - AppStream 9.9 MB/s | 6.7 MB 00:00 2024-03-25T16:14:20.121 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-03-25T16:14:20.445 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-25T16:14:22.211 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - BaseOS 5.6 MB/s | 8.0 MB 00:01 2024-03-25T16:14:22.511 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - BaseOS 3.9 MB/s | 8.0 MB 00:02 2024-03-25T16:14:25.862 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - AppStream 7.2 MB/s | 19 MB 00:02 2024-03-25T16:14:30.079 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - CRB 8.7 MB/s | 6.1 MB 00:00 2024-03-25T16:14:31.579 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 16 kB 00:00 2024-03-25T16:14:33.133 INFO:teuthology.orchestra.run.smithi156.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-25T16:14:38.172 INFO:teuthology.orchestra.run.smithi156.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-25T16:14:40.304 INFO:teuthology.orchestra.run.smithi156.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:14:40.305 INFO:teuthology.orchestra.run.smithi156.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:14:40.405 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:14:40.411 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:14:40.411 INFO:teuthology.orchestra.run.smithi156.stdout: Package Architecture Version Repository Size 2024-03-25T16:14:40.411 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:14:40.411 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T16:14:40.411 INFO:teuthology.orchestra.run.smithi156.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:14:40.412 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-03-25T16:14:40.412 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:14:40.412 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:14:40.412 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:14:40.412 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:14:40.412 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T16:14:40.412 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:14:40.412 INFO:teuthology.orchestra.run.smithi156.stdout:Install 4 Packages 2024-03-25T16:14:40.412 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:14:40.413 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 76 M 2024-03-25T16:14:40.413 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 123 M 2024-03-25T16:14:40.413 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T16:14:40.649 INFO:teuthology.orchestra.run.smithi156.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 333 kB/s | 44 kB 00:00 2024-03-25T16:14:41.266 INFO:teuthology.orchestra.run.smithi156.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 20 MB/s | 15 MB 00:00 2024-03-25T16:14:41.883 INFO:teuthology.orchestra.run.smithi156.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 22 MB/s | 27 MB 00:01 2024-03-25T16:14:42.366 INFO:teuthology.orchestra.run.smithi156.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 18 MB/s | 33 MB 00:01 2024-03-25T16:14:42.366 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:14:42.367 INFO:teuthology.orchestra.run.smithi156.stdout:Total 39 MB/s | 76 MB 00:01 2024-03-25T16:14:43.032 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T16:14:43.427 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T16:14:43.427 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T16:14:44.433 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T16:14:44.433 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T16:14:45.384 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - AppStream 884 kB/s | 19 MB 00:21 2024-03-25T16:14:46.895 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T16:14:47.495 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:14:47.583 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:14:49.368 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-25T16:14:50.362 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:14:50.496 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:14:50.969 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-25T16:14:51.964 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-03-25T16:14:55.267 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:14:55.379 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:14:56.911 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-25T16:14:59.043 INFO:teuthology.orchestra.run.smithi194.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:14:59.044 INFO:teuthology.orchestra.run.smithi194.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:14:59.143 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:14:59.148 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:14:59.148 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-03-25T16:14:59.148 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:14:59.148 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T16:14:59.148 INFO:teuthology.orchestra.run.smithi194.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:14:59.148 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-25T16:14:59.148 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:14:59.148 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:14:59.149 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:14:59.149 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:14:59.149 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T16:14:59.149 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:14:59.149 INFO:teuthology.orchestra.run.smithi194.stdout:Install 4 Packages 2024-03-25T16:14:59.149 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:14:59.149 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 76 M 2024-03-25T16:14:59.150 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 123 M 2024-03-25T16:14:59.150 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T16:14:59.288 INFO:teuthology.orchestra.run.smithi194.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 459 kB/s | 44 kB 00:00 2024-03-25T16:14:59.621 INFO:teuthology.orchestra.run.smithi194.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 35 MB/s | 15 MB 00:00 2024-03-25T16:14:59.963 INFO:teuthology.orchestra.run.smithi194.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 41 MB/s | 27 MB 00:00 2024-03-25T16:15:00.129 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:15:00.255 INFO:teuthology.orchestra.run.smithi194.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 31 MB/s | 33 MB 00:01 2024-03-25T16:15:00.255 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:15:00.255 INFO:teuthology.orchestra.run.smithi194.stdout:Total 69 MB/s | 76 MB 00:01 2024-03-25T16:15:00.923 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T16:15:01.274 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T16:15:01.274 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T16:15:02.296 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T16:15:02.297 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T16:15:04.766 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T16:15:05.394 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:15:05.481 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:15:08.204 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:15:08.340 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:15:13.074 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:15:13.197 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:15:17.921 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:15:29.581 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:15:29.582 INFO:teuthology.orchestra.run.smithi156.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:15:29.582 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:15:29.627 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:15:30.012 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:15:30.012 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:15:30.012 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:15:30.013 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:15:30.421 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:15:30.421 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:15:30.421 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T16:15:30.421 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:15:30.421 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:15:30.421 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:15:30.422 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:15:30.422 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:15:30.422 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:15:30.758 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:15:30.864 INFO:teuthology.orchestra.run.smithi156.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:15:30.864 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:15:30.864 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:15:30.864 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:15:30.864 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:15:30.865 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T16:15:30.865 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi156.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:15:30.865 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:15:30.945 INFO:teuthology.orchestra.run.smithi156.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:15:30.945 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:15:30.945 DEBUG:teuthology.orchestra.run.smithi156:> sudo rpm -qi grub2-tools 2024-03-25T16:15:30.989 INFO:teuthology.orchestra.run.smithi156.stdout:Name : grub2-tools 2024-03-25T16:15:30.990 INFO:teuthology.orchestra.run.smithi156.stdout:Epoch : 1 2024-03-25T16:15:30.990 INFO:teuthology.orchestra.run.smithi156.stdout:Version : 2.06 2024-03-25T16:15:30.990 INFO:teuthology.orchestra.run.smithi156.stdout:Release : 68.el9 2024-03-25T16:15:30.990 INFO:teuthology.orchestra.run.smithi156.stdout:Architecture: x86_64 2024-03-25T16:15:30.990 INFO:teuthology.orchestra.run.smithi156.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:15:30.990 INFO:teuthology.orchestra.run.smithi156.stdout:Group : Unspecified 2024-03-25T16:15:30.990 INFO:teuthology.orchestra.run.smithi156.stdout:Size : 8264254 2024-03-25T16:15:30.990 INFO:teuthology.orchestra.run.smithi156.stdout:License : GPLv3+ 2024-03-25T16:15:30.990 INFO:teuthology.orchestra.run.smithi156.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:15:30.990 INFO:teuthology.orchestra.run.smithi156.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:15:30.990 INFO:teuthology.orchestra.run.smithi156.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:15:30.990 INFO:teuthology.orchestra.run.smithi156.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:15:30.990 INFO:teuthology.orchestra.run.smithi156.stdout:Packager : builder@centos.org 2024-03-25T16:15:30.991 INFO:teuthology.orchestra.run.smithi156.stdout:Vendor : CentOS 2024-03-25T16:15:30.991 INFO:teuthology.orchestra.run.smithi156.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:15:30.991 INFO:teuthology.orchestra.run.smithi156.stdout:Summary : Support tools for GRUB. 2024-03-25T16:15:30.991 INFO:teuthology.orchestra.run.smithi156.stdout:Description : 2024-03-25T16:15:30.991 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:15:30.991 INFO:teuthology.orchestra.run.smithi156.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:15:30.991 INFO:teuthology.orchestra.run.smithi156.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:15:30.991 INFO:teuthology.orchestra.run.smithi156.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:15:30.991 INFO:teuthology.orchestra.run.smithi156.stdout:hardware devices. 2024-03-25T16:15:30.991 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:15:30.991 INFO:teuthology.orchestra.run.smithi156.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:15:30.992 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:15:30.992 INFO:teuthology.task.kernel:Updating grub on smithi156 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:15:30.992 DEBUG:teuthology.orchestra.run.smithi156:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:15:31.571 INFO:teuthology.orchestra.run.smithi156.stderr:Generating grub configuration file ... 2024-03-25T16:15:32.953 INFO:teuthology.orchestra.run.smithi156.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:15:32.987 INFO:teuthology.orchestra.run.smithi156.stderr:done 2024-03-25T16:15:32.989 DEBUG:teuthology.orchestra.run.smithi156:> mktemp 2024-03-25T16:15:33.006 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.5QqZjyA1do 2024-03-25T16:15:33.007 DEBUG:teuthology.orchestra.run.smithi156:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.5QqZjyA1do 2024-03-25T16:15:33.075 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.5QqZjyA1do 2024-03-25T16:15:33.219 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.5QqZjyA1do is 6KB 2024-03-25T16:15:33.267 DEBUG:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.5QqZjyA1do 2024-03-25T16:15:33.282 DEBUG:teuthology.orchestra.run.smithi156:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:15:33.347 INFO:teuthology.orchestra.run.smithi156.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:15:33.347 INFO:teuthology.orchestra.run.smithi156.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:15:33.347 INFO:teuthology.orchestra.run.smithi156.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:15:33.347 INFO:teuthology.orchestra.run.smithi156.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:15:33.349 DEBUG:teuthology.orchestra.run.smithi156:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:15:33.503 DEBUG:teuthology.orchestra.run.smithi156:> sudo shutdown -r now 2024-03-25T16:15:47.232 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:15:47.232 INFO:teuthology.orchestra.run.smithi194.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:15:47.232 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:15:47.276 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:15:47.657 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:15:47.657 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:15:47.657 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:15:47.657 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:15:48.071 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:15:48.072 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:15:48.072 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T16:15:48.072 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:15:48.072 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:15:48.072 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:15:48.072 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:15:48.072 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:15:48.072 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:15:48.388 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:15:48.525 INFO:teuthology.orchestra.run.smithi194.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:15:48.525 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:15:48.525 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:15:48.526 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:15:48.526 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:15:48.526 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T16:15:48.526 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi194.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:15:48.526 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:15:48.606 INFO:teuthology.orchestra.run.smithi194.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:15:48.606 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:15:48.607 DEBUG:teuthology.orchestra.run.smithi194:> sudo rpm -qi grub2-tools 2024-03-25T16:15:48.650 INFO:teuthology.orchestra.run.smithi194.stdout:Name : grub2-tools 2024-03-25T16:15:48.650 INFO:teuthology.orchestra.run.smithi194.stdout:Epoch : 1 2024-03-25T16:15:48.650 INFO:teuthology.orchestra.run.smithi194.stdout:Version : 2.06 2024-03-25T16:15:48.650 INFO:teuthology.orchestra.run.smithi194.stdout:Release : 68.el9 2024-03-25T16:15:48.650 INFO:teuthology.orchestra.run.smithi194.stdout:Architecture: x86_64 2024-03-25T16:15:48.650 INFO:teuthology.orchestra.run.smithi194.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:15:48.650 INFO:teuthology.orchestra.run.smithi194.stdout:Group : Unspecified 2024-03-25T16:15:48.651 INFO:teuthology.orchestra.run.smithi194.stdout:Size : 8264254 2024-03-25T16:15:48.651 INFO:teuthology.orchestra.run.smithi194.stdout:License : GPLv3+ 2024-03-25T16:15:48.651 INFO:teuthology.orchestra.run.smithi194.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:15:48.651 INFO:teuthology.orchestra.run.smithi194.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:15:48.651 INFO:teuthology.orchestra.run.smithi194.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:15:48.651 INFO:teuthology.orchestra.run.smithi194.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:15:48.651 INFO:teuthology.orchestra.run.smithi194.stdout:Packager : builder@centos.org 2024-03-25T16:15:48.651 INFO:teuthology.orchestra.run.smithi194.stdout:Vendor : CentOS 2024-03-25T16:15:48.651 INFO:teuthology.orchestra.run.smithi194.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:15:48.651 INFO:teuthology.orchestra.run.smithi194.stdout:Summary : Support tools for GRUB. 2024-03-25T16:15:48.651 INFO:teuthology.orchestra.run.smithi194.stdout:Description : 2024-03-25T16:15:48.651 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:15:48.651 INFO:teuthology.orchestra.run.smithi194.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:15:48.651 INFO:teuthology.orchestra.run.smithi194.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:15:48.652 INFO:teuthology.orchestra.run.smithi194.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:15:48.652 INFO:teuthology.orchestra.run.smithi194.stdout:hardware devices. 2024-03-25T16:15:48.652 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:15:48.652 INFO:teuthology.orchestra.run.smithi194.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:15:48.653 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:15:48.653 INFO:teuthology.task.kernel:Updating grub on smithi194 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:15:48.653 DEBUG:teuthology.orchestra.run.smithi194:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:15:49.309 INFO:teuthology.orchestra.run.smithi194.stderr:Generating grub configuration file ... 2024-03-25T16:15:50.741 INFO:teuthology.orchestra.run.smithi194.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:15:50.762 INFO:teuthology.orchestra.run.smithi194.stderr:done 2024-03-25T16:15:50.765 DEBUG:teuthology.orchestra.run.smithi194:> mktemp 2024-03-25T16:15:50.820 INFO:teuthology.orchestra.run.smithi194.stdout:/tmp/tmp.rrGzmldjQr 2024-03-25T16:15:50.821 DEBUG:teuthology.orchestra.run.smithi194:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.rrGzmldjQr 2024-03-25T16:15:50.895 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0666 /tmp/tmp.rrGzmldjQr 2024-03-25T16:15:51.033 DEBUG:teuthology.orchestra.remote:smithi194:/tmp/tmp.rrGzmldjQr is 6KB 2024-03-25T16:15:51.082 DEBUG:teuthology.orchestra.run.smithi194:> rm -fr /tmp/tmp.rrGzmldjQr 2024-03-25T16:15:51.096 DEBUG:teuthology.orchestra.run.smithi194:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:15:51.160 INFO:teuthology.orchestra.run.smithi194.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:15:51.161 INFO:teuthology.orchestra.run.smithi194.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:15:51.161 INFO:teuthology.orchestra.run.smithi194.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:15:51.161 INFO:teuthology.orchestra.run.smithi194.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:15:51.162 DEBUG:teuthology.orchestra.run.smithi194:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:15:51.310 DEBUG:teuthology.orchestra.run.smithi194:> sudo shutdown -r now 2024-03-25T16:16:03.518 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T16:16:03.518 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-03-25T16:16:03.519 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:16:21.331 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T16:16:21.331 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-03-25T16:16:21.332 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:16:22.050 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2024-03-25T16:16:31.060 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-03-25T16:16:31.061 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:16:34.146 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2024-03-25T16:16:39.710 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.194 2024-03-25T16:16:46.153 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-03-25T16:16:46.154 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:16:48.714 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-03-25T16:16:48.715 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:16:51.774 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.194 2024-03-25T16:17:03.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-03-25T16:17:03.780 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:17:04.542 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2024-03-25T16:17:19.554 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-03-25T16:17:19.556 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:17:19.987 DEBUG:teuthology.orchestra.run.smithi156:> true 2024-03-25T16:17:20.375 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-03-25T16:17:20.375 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:17:20.375 DEBUG:teuthology.orchestra.run.smithi156:> uname -r 2024-03-25T16:17:20.389 INFO:teuthology.orchestra.run.smithi156.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:17:20.389 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:17:20.390 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:17:20.390 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:17:21.390 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T16:17:21.390 DEBUG:teuthology.orchestra.run.smithi156:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:17:21.462 INFO:teuthology.orchestra.run.smithi156.stdout:ttyS1 2024-03-25T16:17:21.497 DEBUG:teuthology.parallel:result is None 2024-03-25T16:17:35.594 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-03-25T16:17:35.985 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-03-25T16:17:35.985 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:17:35.985 DEBUG:teuthology.orchestra.run.smithi194:> uname -r 2024-03-25T16:17:35.999 INFO:teuthology.orchestra.run.smithi194.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:17:35.999 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:17:35.999 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:17:36.000 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:17:37.001 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T16:17:37.001 DEBUG:teuthology.orchestra.run.smithi194:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:17:37.069 INFO:teuthology.orchestra.run.smithi194.stdout:ttyS1 2024-03-25T16:17:37.106 DEBUG:teuthology.parallel:result is None 2024-03-25T16:17:37.107 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T16:17:37.115 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T16:17:37.115 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:17:37.118 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:17:37.135 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T16:17:37.142 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T16:17:37.150 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T16:17:37.150 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:17:37.177 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:17:37.367 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T16:17:37.374 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T16:17:37.374 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:17:37.377 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:17:37.408 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:17:37.409 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:17:37.422 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:17:37.433 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:17:37.436 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T16:17:37.443 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T16:17:37.443 DEBUG:teuthology.orchestra.run.smithi156:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:17:37.467 DEBUG:teuthology.orchestra.run.smithi194:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:17:37.500 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T16:17:37.509 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T16:17:37.510 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:17:37.537 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:17:37.551 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:17:37.636 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:17:37.723 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:17:37.769 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:17:37.808 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-25T16:17:37.809 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:17:37.875 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:17:37.917 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:17:38.005 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:17:38.045 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:17:38.084 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-25T16:17:38.084 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:17:38.148 DEBUG:teuthology.orchestra.run.smithi156:> sudo service rsyslog restart 2024-03-25T16:17:38.150 DEBUG:teuthology.orchestra.run.smithi194:> sudo service rsyslog restart 2024-03-25T16:17:38.200 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:17:38.459 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:17:38.741 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T16:17:38.750 INFO:teuthology.task.internal:Starting timer... 2024-03-25T16:17:38.750 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T16:17:38.767 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T16:17:38.776 DEBUG:teuthology.orchestra.run.smithi156:> sudo service auditd rotate 2024-03-25T16:17:38.844 INFO:teuthology.orchestra.run.smithi156.stdout:Rotating logs: 2024-03-25T16:17:38.846 DEBUG:teuthology.orchestra.run.smithi194:> sudo service auditd rotate 2024-03-25T16:17:38.955 INFO:teuthology.orchestra.run.smithi194.stdout:Rotating logs: 2024-03-25T16:17:38.959 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T16:17:38.959 DEBUG:teuthology.orchestra.run.smithi156:> /usr/sbin/getenforce 2024-03-25T16:17:38.996 INFO:teuthology.orchestra.run.smithi156.stdout:Permissive 2024-03-25T16:17:38.997 DEBUG:teuthology.orchestra.run.smithi194:> /usr/sbin/getenforce 2024-03-25T16:17:39.031 INFO:teuthology.orchestra.run.smithi194.stdout:Permissive 2024-03-25T16:17:39.032 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi156.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi194.front.sepia.ceph.com': 'permissive'} 2024-03-25T16:17:39.032 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:17:39.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:17:39.064 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:17:39.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:17:39.128 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T16:17:39.128 DEBUG:teuthology.orchestra.run.smithi156:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:17:39.164 DEBUG:teuthology.orchestra.run.smithi194:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:17:39.216 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T16:17:39.226 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-25T16:17:39.232 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-25T16:17:39.368 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T16:17:39.384 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T16:17:39.398 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi156.front.sepia.ceph.com,smithi194.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T16:21:58.046 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi156.front.sepia.ceph.com'), Remote(name='ubuntu@smithi194.front.sepia.ceph.com')] 2024-03-25T16:21:58.047 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-03-25T16:21:58.048 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:21:58.143 DEBUG:teuthology.orchestra.run.smithi156:> true 2024-03-25T16:21:58.225 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-03-25T16:21:58.226 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-03-25T16:21:58.226 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:21:58.356 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-03-25T16:21:58.443 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-03-25T16:21:58.443 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T16:21:58.461 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T16:21:58.461 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:21:58.461 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:21:58.464 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:21:58.464 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:21:58.498 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:21:58.512 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:21:58.516 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:21:58.533 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:21:58.541 INFO:teuthology.orchestra.run.smithi156.stderr:sudo: ntpd: command not found 2024-03-25T16:21:58.555 INFO:teuthology.orchestra.run.smithi156.stdout:506 Cannot talk to daemon 2024-03-25T16:21:58.562 INFO:teuthology.orchestra.run.smithi194.stderr:sudo: ntpd: command not found 2024-03-25T16:21:58.572 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:21:58.576 INFO:teuthology.orchestra.run.smithi194.stdout:506 Cannot talk to daemon 2024-03-25T16:21:58.588 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:21:58.593 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:21:58.609 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:21:58.641 INFO:teuthology.orchestra.run.smithi156.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:21:58.644 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:21:58.644 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2024-03-25T16:21:58.644 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:21:58.644 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:21:58.644 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:21:58.644 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:21:58.659 INFO:teuthology.orchestra.run.smithi194.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:21:58.662 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:21:58.662 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-03-25T16:21:58.662 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:21:58.663 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:21:58.663 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:21:58.663 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:21:58.664 INFO:teuthology.run_tasks:Running task install... 2024-03-25T16:21:58.672 DEBUG:teuthology.task.install:project ceph 2024-03-25T16:21:58.672 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T16:21:58.672 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T16:21:58.672 INFO:teuthology.task.install:Using flavor: default 2024-03-25T16:21:58.678 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T16:21:58.678 INFO:teuthology.task.install:extra packages: [] 2024-03-25T16:21:58.678 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:21:58.679 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:21:58.681 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:21:58.681 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:21:58.840 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:21:58.841 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:21:58.915 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:21:58.915 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:21:59.173 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:21:59.173 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-25T16:21:59.174 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:21:59.202 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:21:59.202 DEBUG:teuthology.orchestra.run.smithi194:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:21:59.251 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:21:59.251 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-25T16:21:59.252 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:21:59.270 DEBUG:teuthology.orchestra.run.smithi194:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:21:59.281 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:21:59.281 DEBUG:teuthology.orchestra.run.smithi156:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:21:59.351 DEBUG:teuthology.orchestra.run.smithi194:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:21:59.352 DEBUG:teuthology.orchestra.run.smithi156:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:21:59.417 INFO:teuthology.orchestra.run.smithi194.stdout:check_obsoletes = 1 2024-03-25T16:21:59.418 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum clean all 2024-03-25T16:21:59.433 DEBUG:teuthology.orchestra.run.smithi156:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:21:59.501 INFO:teuthology.orchestra.run.smithi156.stdout:check_obsoletes = 1 2024-03-25T16:21:59.504 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum clean all 2024-03-25T16:21:59.661 INFO:teuthology.orchestra.run.smithi194.stdout:74 files removed 2024-03-25T16:21:59.691 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-radosgw 2024-03-25T16:21:59.743 INFO:teuthology.orchestra.run.smithi156.stdout:74 files removed 2024-03-25T16:21:59.771 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-radosgw 2024-03-25T16:22:00.986 INFO:teuthology.orchestra.run.smithi156.stdout:ceph packages for x86_64 90 kB/s | 82 kB 00:00 2024-03-25T16:22:01.347 INFO:teuthology.orchestra.run.smithi194.stdout:ceph packages for x86_64 60 kB/s | 82 kB 00:01 2024-03-25T16:22:01.877 INFO:teuthology.orchestra.run.smithi156.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T16:22:02.139 INFO:teuthology.orchestra.run.smithi194.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-25T16:22:02.661 INFO:teuthology.orchestra.run.smithi156.stdout:ceph source packages 2.5 kB/s | 1.9 kB 00:00 2024-03-25T16:22:03.022 INFO:teuthology.orchestra.run.smithi194.stdout:ceph source packages 2.2 kB/s | 1.9 kB 00:00 2024-03-25T16:22:03.059 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS-9 - AppStream 19 MB/s | 6.7 MB 00:00 2024-03-25T16:22:03.433 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-25T16:22:04.473 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-03-25T16:22:04.834 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-25T16:22:05.288 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-03-25T16:22:05.924 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-25T16:22:07.642 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-03-25T16:22:19.334 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - CRB 761 kB/s | 6.1 MB 00:08 2024-03-25T16:22:20.842 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 16 kB 00:00 2024-03-25T16:22:23.110 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 9.6 MB/s | 21 MB 00:02 2024-03-25T16:22:27.528 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - AppStream 910 kB/s | 19 MB 00:21 2024-03-25T16:22:28.077 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-25T16:22:30.522 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:22:30.539 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:22:30.539 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-25T16:22:30.539 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:22:30.539 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T16:22:30.539 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:22:30.539 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrading: 2024-03-25T16:22:30.539 INFO:teuthology.orchestra.run.smithi194.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:22:30.539 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:22:30.539 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-25T16:22:30.540 INFO:teuthology.orchestra.run.smithi194.stdout: boost-program-options 2024-03-25T16:22:30.540 INFO:teuthology.orchestra.run.smithi194.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:22:30.540 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:22:30.540 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:22:30.540 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:22:30.540 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:22:30.540 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:22:30.540 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:22:30.540 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:22:30.540 INFO:teuthology.orchestra.run.smithi194.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:22:30.540 INFO:teuthology.orchestra.run.smithi194.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:22:30.541 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:22:30.541 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:22:30.541 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:22:30.541 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:22:30.541 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:22:30.541 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:22:30.541 INFO:teuthology.orchestra.run.smithi194.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:22:30.541 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse 2024-03-25T16:22:30.541 INFO:teuthology.orchestra.run.smithi194.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:22:30.541 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:22:30.541 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:22:30.541 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:22:30.542 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:22:30.542 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:22:30.542 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:22:30.542 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:22:30.542 INFO:teuthology.orchestra.run.smithi194.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:22:30.542 INFO:teuthology.orchestra.run.smithi194.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:22:30.542 INFO:teuthology.orchestra.run.smithi194.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:22:30.542 INFO:teuthology.orchestra.run.smithi194.stdout:Installing weak dependencies: 2024-03-25T16:22:30.542 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:22:30.542 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:22:30.542 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T16:22:30.542 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:22:30.543 INFO:teuthology.orchestra.run.smithi194.stdout:Install 30 Packages 2024-03-25T16:22:30.543 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrade 2 Packages 2024-03-25T16:22:30.543 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:22:30.543 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 58 M 2024-03-25T16:22:30.543 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T16:22:31.554 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-25T16:22:33.879 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 16 kB 00:01 2024-03-25T16:22:34.581 INFO:teuthology.orchestra.run.smithi156.stdout:Extra Packages for Enterprise Linux 35 MB/s | 21 MB 00:00 2024-03-25T16:22:38.686 INFO:teuthology.orchestra.run.smithi194.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 691 kB/s | 5.3 MB 00:07 2024-03-25T16:22:38.777 INFO:teuthology.orchestra.run.smithi194.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 275 kB/s | 25 kB 00:00 2024-03-25T16:22:39.489 INFO:teuthology.orchestra.run.smithi156.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-25T16:22:40.119 INFO:teuthology.orchestra.run.smithi194.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 553 kB/s | 742 kB 00:01 2024-03-25T16:22:41.053 INFO:teuthology.orchestra.run.smithi194.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 534 kB/s | 498 kB 00:00 2024-03-25T16:22:41.909 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:22:41.911 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:22:41.912 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-03-25T16:22:41.912 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:22:41.912 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T16:22:41.912 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:22:41.912 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrading: 2024-03-25T16:22:41.912 INFO:teuthology.orchestra.run.smithi156.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:22:41.912 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:22:41.913 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-03-25T16:22:41.913 INFO:teuthology.orchestra.run.smithi156.stdout: boost-program-options 2024-03-25T16:22:41.913 INFO:teuthology.orchestra.run.smithi156.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:22:41.913 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:22:41.913 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:22:41.913 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:22:41.913 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:22:41.913 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:22:41.914 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:22:41.914 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:22:41.914 INFO:teuthology.orchestra.run.smithi156.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:22:41.914 INFO:teuthology.orchestra.run.smithi156.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:22:41.914 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:22:41.914 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:22:41.914 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:22:41.915 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:22:41.915 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:22:41.915 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:22:41.915 INFO:teuthology.orchestra.run.smithi156.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:22:41.915 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse 2024-03-25T16:22:41.915 INFO:teuthology.orchestra.run.smithi156.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:22:41.915 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:22:41.916 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:22:41.916 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:22:41.916 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:22:41.916 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:22:41.916 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:22:41.916 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:22:41.916 INFO:teuthology.orchestra.run.smithi156.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:22:41.916 INFO:teuthology.orchestra.run.smithi156.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:22:41.917 INFO:teuthology.orchestra.run.smithi156.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:22:41.917 INFO:teuthology.orchestra.run.smithi156.stdout:Installing weak dependencies: 2024-03-25T16:22:41.917 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:22:41.919 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:22:41.919 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T16:22:41.919 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:22:41.919 INFO:teuthology.orchestra.run.smithi156.stdout:Install 30 Packages 2024-03-25T16:22:41.919 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrade 2 Packages 2024-03-25T16:22:41.919 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:22:41.920 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 58 M 2024-03-25T16:22:41.920 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T16:22:47.819 INFO:teuthology.orchestra.run.smithi194.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 772 kB/s | 5.1 MB 00:06 2024-03-25T16:22:47.952 INFO:teuthology.orchestra.run.smithi194.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 339 kB/s | 45 kB 00:00 2024-03-25T16:22:48.136 INFO:teuthology.orchestra.run.smithi194.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 717 kB/s | 131 kB 00:00 2024-03-25T16:22:48.402 INFO:teuthology.orchestra.run.smithi194.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 607 kB/s | 162 kB 00:00 2024-03-25T16:22:48.886 INFO:teuthology.orchestra.run.smithi194.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 663 kB/s | 320 kB 00:00 2024-03-25T16:22:49.353 INFO:teuthology.orchestra.run.smithi194.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 638 kB/s | 298 kB 00:00 2024-03-25T16:22:49.528 INFO:teuthology.orchestra.run.smithi194.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 568 kB/s | 99 kB 00:00 2024-03-25T16:22:49.637 INFO:teuthology.orchestra.run.smithi194.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 979 kB/s | 106 kB 00:00 2024-03-25T16:22:49.679 INFO:teuthology.orchestra.run.smithi194.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-25T16:22:49.712 INFO:teuthology.orchestra.run.smithi194.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T16:22:49.771 INFO:teuthology.orchestra.run.smithi194.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T16:22:49.813 INFO:teuthology.orchestra.run.smithi194.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T16:22:49.846 INFO:teuthology.orchestra.run.smithi194.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T16:22:49.988 INFO:teuthology.orchestra.run.smithi194.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.5 MB/s | 216 kB 00:00 2024-03-25T16:22:50.030 INFO:teuthology.orchestra.run.smithi194.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 843 kB/s | 35 kB 00:00 2024-03-25T16:22:50.072 INFO:teuthology.orchestra.run.smithi194.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-25T16:22:50.322 INFO:teuthology.orchestra.run.smithi194.stdout:(21/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 510 kB/s | 9.7 MB 00:19 2024-03-25T16:22:50.381 INFO:teuthology.orchestra.run.smithi194.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.0 MB/s | 309 kB 00:00 2024-03-25T16:22:50.440 INFO:teuthology.orchestra.run.smithi194.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 445 kB/s | 26 kB 00:00 2024-03-25T16:22:50.507 INFO:teuthology.orchestra.run.smithi194.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 728 kB/s | 49 kB 00:00 2024-03-25T16:22:50.582 INFO:teuthology.orchestra.run.smithi194.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 900 kB/s | 67 kB 00:00 2024-03-25T16:22:50.749 INFO:teuthology.orchestra.run.smithi194.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-03-25T16:22:50.807 INFO:teuthology.orchestra.run.smithi194.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-25T16:22:50.841 INFO:teuthology.orchestra.run.smithi194.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.2 MB/s | 838 kB 00:00 2024-03-25T16:22:50.883 INFO:teuthology.orchestra.run.smithi194.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-03-25T16:22:55.908 INFO:teuthology.orchestra.run.smithi194.stdout:(30/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 810 kB/s | 20 MB 00:25 2024-03-25T16:22:55.964 INFO:teuthology.orchestra.run.smithi156.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 717 kB/s | 9.7 MB 00:13 2024-03-25T16:22:56.072 INFO:teuthology.orchestra.run.smithi156.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 233 kB/s | 25 kB 00:00 2024-03-25T16:22:56.558 INFO:teuthology.orchestra.run.smithi194.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 604 kB/s | 3.4 MB 00:05 2024-03-25T16:22:56.930 INFO:teuthology.orchestra.run.smithi156.stdout:(3/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 368 kB/s | 5.3 MB 00:14 2024-03-25T16:22:57.058 INFO:teuthology.orchestra.run.smithi194.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 519 kB/s | 3.1 MB 00:06 2024-03-25T16:22:57.061 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:57.061 INFO:teuthology.orchestra.run.smithi194.stdout:Total 2.2 MB/s | 58 MB 00:26 2024-03-25T16:22:57.094 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T16:22:57.131 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T16:22:57.131 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T16:22:57.306 INFO:teuthology.orchestra.run.smithi156.stdout:(4/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 601 kB/s | 742 kB 00:01 2024-03-25T16:22:57.596 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T16:22:57.596 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T16:22:57.973 INFO:teuthology.orchestra.run.smithi156.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 478 kB/s | 498 kB 00:01 2024-03-25T16:22:58.074 INFO:teuthology.orchestra.run.smithi156.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 452 kB/s | 45 kB 00:00 2024-03-25T16:22:58.365 INFO:teuthology.orchestra.run.smithi156.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 450 kB/s | 131 kB 00:00 2024-03-25T16:22:58.665 INFO:teuthology.orchestra.run.smithi156.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 540 kB/s | 162 kB 00:00 2024-03-25T16:22:59.094 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T16:22:59.170 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:22:59.204 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:22:59.230 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:22:59.252 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:22:59.269 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:22:59.349 INFO:teuthology.orchestra.run.smithi156.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 469 kB/s | 320 kB 00:00 2024-03-25T16:22:59.462 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:22:59.516 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:22:59.907 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:22:59.948 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:23:00.032 INFO:teuthology.orchestra.run.smithi156.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 436 kB/s | 298 kB 00:00 2024-03-25T16:23:00.057 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:23:00.144 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:23:00.166 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:23:00.229 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:23:00.241 INFO:teuthology.orchestra.run.smithi156.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 477 kB/s | 99 kB 00:00 2024-03-25T16:23:00.257 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:23:00.306 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:23:00.368 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:23:00.393 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:23:00.433 INFO:teuthology.orchestra.run.smithi156.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 554 kB/s | 106 kB 00:00 2024-03-25T16:23:00.456 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:23:00.491 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:23:00.492 INFO:teuthology.orchestra.run.smithi156.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-25T16:23:00.525 INFO:teuthology.orchestra.run.smithi156.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T16:23:00.583 INFO:teuthology.orchestra.run.smithi156.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T16:23:00.599 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:23:00.625 INFO:teuthology.orchestra.run.smithi156.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T16:23:00.651 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:23:00.659 INFO:teuthology.orchestra.run.smithi156.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T16:23:00.692 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:23:00.723 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:23:00.769 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:23:00.802 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:23:00.868 INFO:teuthology.orchestra.run.smithi156.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.0 MB/s | 216 kB 00:00 2024-03-25T16:23:00.918 INFO:teuthology.orchestra.run.smithi156.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 700 kB/s | 35 kB 00:00 2024-03-25T16:23:00.977 INFO:teuthology.orchestra.run.smithi156.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 762 kB/s | 44 kB 00:00 2024-03-25T16:23:01.111 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:23:01.140 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:23:01.213 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:23:01.244 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:23:01.290 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:23:01.485 INFO:teuthology.orchestra.run.smithi156.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 608 kB/s | 309 kB 00:00 2024-03-25T16:23:01.758 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:23:01.784 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:23:01.856 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:23:01.932 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:23:01.961 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:23:02.527 INFO:teuthology.orchestra.run.smithi156.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.2 MB/s | 4.4 MB 00:01 2024-03-25T16:23:02.795 INFO:teuthology.orchestra.run.smithi156.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 97 kB/s | 26 kB 00:00 2024-03-25T16:23:03.144 INFO:teuthology.orchestra.run.smithi156.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 139 kB/s | 49 kB 00:00 2024-03-25T16:23:03.486 INFO:teuthology.orchestra.run.smithi156.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 197 kB/s | 67 kB 00:00 2024-03-25T16:23:04.127 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:23:04.169 INFO:teuthology.orchestra.run.smithi156.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.2 MB/s | 838 kB 00:00 2024-03-25T16:23:04.174 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:23:04.611 INFO:teuthology.orchestra.run.smithi156.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 434 kB/s | 192 kB 00:00 2024-03-25T16:23:04.735 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:23:04.769 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:23:04.858 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:23:04.859 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:23:04.859 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:23:04.859 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:23:04.903 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:23:05.270 INFO:teuthology.orchestra.run.smithi156.stdout:(28/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 656 kB/s | 5.1 MB 00:07 2024-03-25T16:23:05.312 INFO:teuthology.orchestra.run.smithi156.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 2.3 MB/s | 1.6 MB 00:00 2024-03-25T16:23:10.712 INFO:teuthology.orchestra.run.smithi156.stdout:(30/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 711 kB/s | 20 MB 00:28 2024-03-25T16:23:11.203 INFO:teuthology.orchestra.run.smithi156.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 544 kB/s | 3.1 MB 00:05 2024-03-25T16:23:11.879 INFO:teuthology.orchestra.run.smithi156.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 523 kB/s | 3.4 MB 00:06 2024-03-25T16:23:11.883 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:11.883 INFO:teuthology.orchestra.run.smithi156.stdout:Total 1.9 MB/s | 58 MB 00:29 2024-03-25T16:23:11.914 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T16:23:11.952 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T16:23:11.952 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T16:23:12.426 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T16:23:12.426 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T16:23:13.912 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T16:23:14.005 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:23:14.050 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:23:14.083 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:23:14.107 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:23:14.123 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:23:14.152 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:23:14.152 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /sys 2024-03-25T16:23:14.152 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /proc 2024-03-25T16:23:14.152 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /mnt 2024-03-25T16:23:14.152 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /var/tmp 2024-03-25T16:23:14.153 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /home 2024-03-25T16:23:14.153 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /root 2024-03-25T16:23:14.153 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /tmp 2024-03-25T16:23:14.153 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:23:14.314 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:23:14.370 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:23:14.689 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:23:14.769 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:23:14.851 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:23:14.939 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:23:14.970 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:23:15.041 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:23:15.070 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:23:15.070 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:23:15.094 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:23:15.094 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:15.094 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:23:15.094 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:23:15.094 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:23:15.095 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:23:15.096 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:23:15.113 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:23:15.181 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:23:15.189 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:23:15.189 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:23:15.206 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:23:15.251 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:23:15.295 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:23:15.404 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:23:15.447 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:23:15.487 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:23:15.520 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:23:15.548 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:23:15.581 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:23:15.890 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:23:15.919 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:23:16.001 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:23:16.033 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:23:16.071 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:23:16.271 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:23:16.271 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:23:16.272 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:23:16.272 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:23:16.272 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:23:16.272 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:23:16.272 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:23:16.272 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:23:16.272 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:23:16.272 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:23:16.272 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:23:16.272 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:23:16.272 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:23:16.272 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:23:16.273 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:23:16.273 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:23:16.273 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:23:16.273 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:23:16.273 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:23:16.273 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:23:16.273 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:23:16.273 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:23:16.273 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:23:16.273 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:23:16.273 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:23:16.273 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:23:16.274 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:23:16.274 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:23:16.274 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:23:16.274 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:23:16.274 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:23:16.274 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:23:16.274 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:23:16.274 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:23:16.526 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:23:16.555 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:23:16.627 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:23:16.694 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:23:16.733 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:23:17.366 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:23:17.366 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:23:17.366 INFO:teuthology.orchestra.run.smithi194.stdout:Upgraded: 2024-03-25T16:23:17.366 INFO:teuthology.orchestra.run.smithi194.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:17.366 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:17.366 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T16:23:17.366 INFO:teuthology.orchestra.run.smithi194.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:23:17.367 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:17.367 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:17.367 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:17.367 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:17.367 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:23:17.367 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:23:17.367 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:23:17.367 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:17.367 INFO:teuthology.orchestra.run.smithi194.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:23:17.367 INFO:teuthology.orchestra.run.smithi194.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:23:17.367 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:23:17.367 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:17.367 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:23:17.368 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:17.368 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:23:17.368 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:23:17.368 INFO:teuthology.orchestra.run.smithi194.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:23:17.368 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:17.368 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:17.368 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:17.368 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:23:17.368 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:17.368 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:17.368 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:17.368 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:23:17.368 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:23:17.369 INFO:teuthology.orchestra.run.smithi194.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:23:17.369 INFO:teuthology.orchestra.run.smithi194.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:23:17.369 INFO:teuthology.orchestra.run.smithi194.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:23:17.369 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:23:17.369 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:23:17.737 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-test 2024-03-25T16:23:18.297 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:50 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:23:18.446 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:23:18.447 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:23:18.447 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-25T16:23:18.447 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:23:18.447 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T16:23:18.447 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:23:18.447 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-25T16:23:18.447 INFO:teuthology.orchestra.run.smithi194.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:23:18.447 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:23:18.448 INFO:teuthology.orchestra.run.smithi194.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:23:18.448 INFO:teuthology.orchestra.run.smithi194.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:23:18.448 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:23:18.448 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:23:18.448 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T16:23:18.448 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:23:18.448 INFO:teuthology.orchestra.run.smithi194.stdout:Install 6 Packages 2024-03-25T16:23:18.448 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:23:18.449 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 48 M 2024-03-25T16:23:18.449 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 204 M 2024-03-25T16:23:18.449 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T16:23:18.895 INFO:teuthology.orchestra.run.smithi194.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-25T16:23:18.920 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:23:18.945 INFO:teuthology.orchestra.run.smithi194.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-25T16:23:18.962 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:23:18.979 INFO:teuthology.orchestra.run.smithi194.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-25T16:23:19.487 INFO:teuthology.orchestra.run.smithi194.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 368 kB/s | 187 kB 00:00 2024-03-25T16:23:19.503 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:23:19.541 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:23:19.595 INFO:teuthology.orchestra.run.smithi194.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 188 kB/s | 163 kB 00:00 2024-03-25T16:23:19.634 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:23:19.634 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:23:19.634 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:23:19.634 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:23:19.692 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:23:29.163 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:23:29.163 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /sys 2024-03-25T16:23:29.163 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /proc 2024-03-25T16:23:29.163 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /mnt 2024-03-25T16:23:29.163 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /var/tmp 2024-03-25T16:23:29.163 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /home 2024-03-25T16:23:29.163 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /root 2024-03-25T16:23:29.163 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /tmp 2024-03-25T16:23:29.163 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:23:30.060 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:23:30.084 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:23:30.084 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:30.084 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:23:30.084 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:23:30.084 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:23:30.084 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:23:30.086 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:23:30.154 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:23:30.155 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:23:31.271 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:23:31.271 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:23:31.271 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:23:31.272 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:23:31.272 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:23:31.272 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:23:31.272 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:23:31.272 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:23:31.272 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:23:31.272 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:23:31.272 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:23:31.272 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:23:31.272 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:23:31.273 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:23:31.273 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:23:31.273 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:23:31.273 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:23:31.273 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:23:31.273 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:23:31.273 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:23:31.273 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:23:31.273 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:23:31.273 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:23:31.273 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:23:31.273 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:23:31.273 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:23:31.274 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:23:31.274 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:23:31.275 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:23:31.275 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:23:31.275 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:23:31.275 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:23:31.275 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:23:31.275 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:23:32.361 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:23:32.361 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:23:32.361 INFO:teuthology.orchestra.run.smithi156.stdout:Upgraded: 2024-03-25T16:23:32.361 INFO:teuthology.orchestra.run.smithi156.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:32.362 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:32.362 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T16:23:32.362 INFO:teuthology.orchestra.run.smithi156.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:23:32.362 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:32.362 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:32.362 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:32.362 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:32.362 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:23:32.362 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:23:32.362 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:23:32.362 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:32.362 INFO:teuthology.orchestra.run.smithi156.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:23:32.363 INFO:teuthology.orchestra.run.smithi156.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:23:32.363 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:23:32.363 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:32.363 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:23:32.363 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:32.363 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:23:32.363 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:23:32.363 INFO:teuthology.orchestra.run.smithi156.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:23:32.363 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:32.363 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:32.363 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:32.363 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:23:32.363 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:32.363 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:32.364 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:32.364 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:23:32.364 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:23:32.364 INFO:teuthology.orchestra.run.smithi156.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:23:32.364 INFO:teuthology.orchestra.run.smithi156.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:23:32.364 INFO:teuthology.orchestra.run.smithi156.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:23:32.364 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:23:32.364 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:23:32.687 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-test 2024-03-25T16:23:33.276 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:00:54 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:23:33.426 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:23:33.427 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:23:33.427 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-03-25T16:23:33.427 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:23:33.427 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T16:23:33.427 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:23:33.427 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-03-25T16:23:33.427 INFO:teuthology.orchestra.run.smithi156.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:23:33.428 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:23:33.428 INFO:teuthology.orchestra.run.smithi156.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:23:33.428 INFO:teuthology.orchestra.run.smithi156.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:23:33.428 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:23:33.428 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:23:33.428 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T16:23:33.428 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:23:33.428 INFO:teuthology.orchestra.run.smithi156.stdout:Install 6 Packages 2024-03-25T16:23:33.428 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:23:33.429 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 48 M 2024-03-25T16:23:33.429 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 204 M 2024-03-25T16:23:33.429 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T16:23:33.725 INFO:teuthology.orchestra.run.smithi156.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-25T16:23:33.775 INFO:teuthology.orchestra.run.smithi156.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-25T16:23:33.816 INFO:teuthology.orchestra.run.smithi156.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-25T16:23:33.950 INFO:teuthology.orchestra.run.smithi156.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-25T16:23:34.333 INFO:teuthology.orchestra.run.smithi156.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 213 kB/s | 163 kB 00:00 2024-03-25T16:24:28.611 INFO:teuthology.orchestra.run.smithi156.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 879 kB/s | 47 MB 00:55 2024-03-25T16:24:28.612 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:28.613 INFO:teuthology.orchestra.run.smithi156.stdout:Total 894 kB/s | 48 MB 00:55 2024-03-25T16:24:28.625 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T16:24:28.646 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T16:24:28.646 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T16:24:29.041 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T16:24:29.042 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T16:24:29.627 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T16:24:29.710 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:24:29.754 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:24:29.778 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:24:29.827 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:24:29.854 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:24:34.418 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:24:34.498 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:24:36.941 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:24:36.941 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:24:36.941 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:24:36.941 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:24:36.941 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:24:36.941 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:24:37.323 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:24:37.323 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:24:37.323 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T16:24:37.323 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:37.323 INFO:teuthology.orchestra.run.smithi156.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:24:37.323 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:37.323 INFO:teuthology.orchestra.run.smithi156.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:24:37.323 INFO:teuthology.orchestra.run.smithi156.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:24:37.323 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:24:37.323 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:24:37.323 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:24:37.649 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph 2024-03-25T16:24:38.222 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:01:59 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:24:38.383 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:24:38.385 INFO:teuthology.orchestra.run.smithi156.stdout:========================================================================================= 2024-03-25T16:24:38.385 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-03-25T16:24:38.385 INFO:teuthology.orchestra.run.smithi156.stdout:========================================================================================= 2024-03-25T16:24:38.386 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T16:24:38.386 INFO:teuthology.orchestra.run.smithi156.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:24:38.386 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-03-25T16:24:38.386 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:24:38.386 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:24:38.386 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:24:38.386 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:24:38.386 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:24:38.386 INFO:teuthology.orchestra.run.smithi156.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:24:38.386 INFO:teuthology.orchestra.run.smithi156.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:24:38.386 INFO:teuthology.orchestra.run.smithi156.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:24:38.386 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:24:38.386 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:24:38.387 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:24:38.387 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:24:38.387 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:24:38.387 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:24:38.387 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:24:38.387 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:24:38.387 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:24:38.387 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:24:38.387 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:24:38.387 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:24:38.387 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:24:38.387 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:24:38.388 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:24:38.388 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:24:38.388 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:24:38.388 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:24:38.388 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:24:38.388 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:24:38.388 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:24:38.388 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:24:38.388 INFO:teuthology.orchestra.run.smithi156.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:24:38.388 INFO:teuthology.orchestra.run.smithi156.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:24:38.388 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:24:38.388 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:24:38.388 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:24:38.389 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:24:38.389 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T16:24:38.389 INFO:teuthology.orchestra.run.smithi156.stdout:========================================================================================= 2024-03-25T16:24:38.389 INFO:teuthology.orchestra.run.smithi156.stdout:Install 36 Packages 2024-03-25T16:24:38.389 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:24:38.389 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 30 M 2024-03-25T16:24:38.390 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 107 M 2024-03-25T16:24:38.390 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T16:24:39.240 INFO:teuthology.orchestra.run.smithi156.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 16 kB/s | 6.6 kB 00:00 2024-03-25T16:24:45.190 INFO:teuthology.orchestra.run.smithi156.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 239 kB/s | 1.5 MB 00:06 2024-03-25T16:24:45.498 INFO:teuthology.orchestra.run.smithi156.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 325 kB/s | 2.1 MB 00:06 2024-03-25T16:24:46.699 INFO:teuthology.orchestra.run.smithi156.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 206 kB/s | 247 kB 00:01 2024-03-25T16:24:46.832 INFO:teuthology.orchestra.run.smithi156.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-25T16:24:46.874 INFO:teuthology.orchestra.run.smithi156.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T16:24:46.908 INFO:teuthology.orchestra.run.smithi156.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 948 kB/s | 32 kB 00:00 2024-03-25T16:24:46.941 INFO:teuthology.orchestra.run.smithi156.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T16:24:46.975 INFO:teuthology.orchestra.run.smithi156.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T16:24:47.017 INFO:teuthology.orchestra.run.smithi156.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T16:24:47.084 INFO:teuthology.orchestra.run.smithi156.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-25T16:24:47.126 INFO:teuthology.orchestra.run.smithi156.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-25T16:24:47.159 INFO:teuthology.orchestra.run.smithi156.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T16:24:47.193 INFO:teuthology.orchestra.run.smithi156.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-25T16:24:47.385 INFO:teuthology.orchestra.run.smithi156.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 652 kB/s | 125 kB 00:00 2024-03-25T16:24:47.469 INFO:teuthology.orchestra.run.smithi156.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2024-03-25T16:24:47.686 INFO:teuthology.orchestra.run.smithi156.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 697 kB/s | 151 kB 00:00 2024-03-25T16:24:47.744 INFO:teuthology.orchestra.run.smithi156.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 746 kB/s | 43 kB 00:00 2024-03-25T16:24:47.845 INFO:teuthology.orchestra.run.smithi156.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-03-25T16:24:47.945 INFO:teuthology.orchestra.run.smithi156.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2024-03-25T16:24:48.003 INFO:teuthology.orchestra.run.smithi156.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 184 kB/s | 11 kB 00:00 2024-03-25T16:24:48.062 INFO:teuthology.orchestra.run.smithi156.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 302 kB/s | 18 kB 00:00 2024-03-25T16:24:48.121 INFO:teuthology.orchestra.run.smithi156.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 397 kB/s | 23 kB 00:00 2024-03-25T16:24:48.179 INFO:teuthology.orchestra.run.smithi156.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 334 kB/s | 19 kB 00:00 2024-03-25T16:24:48.238 INFO:teuthology.orchestra.run.smithi156.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 337 kB/s | 20 kB 00:00 2024-03-25T16:24:48.322 INFO:teuthology.orchestra.run.smithi156.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 555 kB/s | 46 kB 00:00 2024-03-25T16:24:48.405 INFO:teuthology.orchestra.run.smithi156.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 948 kB/s | 79 kB 00:00 2024-03-25T16:24:48.506 INFO:teuthology.orchestra.run.smithi156.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-03-25T16:24:48.573 INFO:teuthology.orchestra.run.smithi156.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 246 kB/s | 16 kB 00:00 2024-03-25T16:24:48.648 INFO:teuthology.orchestra.run.smithi156.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-25T16:24:48.707 INFO:teuthology.orchestra.run.smithi156.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 615 kB/s | 36 kB 00:00 2024-03-25T16:24:48.790 INFO:teuthology.orchestra.run.smithi156.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-25T16:24:48.882 INFO:teuthology.orchestra.run.smithi156.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-03-25T16:24:48.958 INFO:teuthology.orchestra.run.smithi156.stdout:(34/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 494 kB/s | 4.7 MB 00:09 2024-03-25T16:24:48.983 INFO:teuthology.orchestra.run.smithi156.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2024-03-25T16:24:52.933 INFO:teuthology.orchestra.run.smithi194.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 514 kB/s | 47 MB 01:34 2024-03-25T16:24:52.935 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:52.935 INFO:teuthology.orchestra.run.smithi194.stdout:Total 522 kB/s | 48 MB 01:34 2024-03-25T16:24:52.948 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T16:24:52.968 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T16:24:52.968 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T16:24:53.348 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T16:24:53.349 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T16:24:53.934 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T16:24:54.033 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:24:54.092 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:24:54.132 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:24:54.193 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:24:54.219 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:24:58.865 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:24:58.951 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:25:01.223 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:25:01.223 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:25:01.223 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:25:01.223 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:25:01.223 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:25:01.223 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:25:01.580 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:25:01.638 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:25:01.639 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T16:25:01.639 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:01.639 INFO:teuthology.orchestra.run.smithi194.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:25:01.639 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:01.639 INFO:teuthology.orchestra.run.smithi194.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:25:01.639 INFO:teuthology.orchestra.run.smithi194.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:25:01.639 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:25:01.639 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:25:01.639 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:25:01.875 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph 2024-03-25T16:25:02.451 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:34 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:25:02.610 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:25:02.613 INFO:teuthology.orchestra.run.smithi194.stdout:========================================================================================= 2024-03-25T16:25:02.613 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-25T16:25:02.613 INFO:teuthology.orchestra.run.smithi194.stdout:========================================================================================= 2024-03-25T16:25:02.613 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T16:25:02.613 INFO:teuthology.orchestra.run.smithi194.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:25:02.613 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-25T16:25:02.613 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:25:02.613 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:25:02.613 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:25:02.613 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:25:02.613 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:25:02.613 INFO:teuthology.orchestra.run.smithi194.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:25:02.614 INFO:teuthology.orchestra.run.smithi194.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:25:02.614 INFO:teuthology.orchestra.run.smithi194.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:25:02.614 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:25:02.614 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:25:02.614 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:25:02.614 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:25:02.614 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:25:02.614 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:25:02.614 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:25:02.614 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:25:02.614 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:25:02.614 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:25:02.614 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:25:02.615 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:25:02.615 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:25:02.615 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:25:02.615 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:25:02.615 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:25:02.615 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:25:02.615 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:25:02.615 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:25:02.615 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:25:02.615 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:25:02.615 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:25:02.615 INFO:teuthology.orchestra.run.smithi194.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:25:02.615 INFO:teuthology.orchestra.run.smithi194.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:25:02.616 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:25:02.616 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:25:02.616 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:25:02.616 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:25:02.616 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T16:25:02.616 INFO:teuthology.orchestra.run.smithi194.stdout:========================================================================================= 2024-03-25T16:25:02.616 INFO:teuthology.orchestra.run.smithi194.stdout:Install 36 Packages 2024-03-25T16:25:02.616 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:25:02.617 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 30 M 2024-03-25T16:25:02.617 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 107 M 2024-03-25T16:25:02.617 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T16:25:03.368 INFO:teuthology.orchestra.run.smithi194.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 15 kB/s | 6.6 kB 00:00 2024-03-25T16:25:06.493 INFO:teuthology.orchestra.run.smithi194.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 427 kB/s | 1.5 MB 00:03 2024-03-25T16:25:09.901 INFO:teuthology.orchestra.run.smithi194.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 311 kB/s | 2.1 MB 00:06 2024-03-25T16:25:10.584 INFO:teuthology.orchestra.run.smithi194.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 362 kB/s | 247 kB 00:00 2024-03-25T16:25:10.734 INFO:teuthology.orchestra.run.smithi194.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-25T16:25:10.793 INFO:teuthology.orchestra.run.smithi194.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-25T16:25:10.819 INFO:teuthology.orchestra.run.smithi194.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-25T16:25:10.843 INFO:teuthology.orchestra.run.smithi194.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-03-25T16:25:10.869 INFO:teuthology.orchestra.run.smithi194.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-25T16:25:10.919 INFO:teuthology.orchestra.run.smithi194.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-25T16:25:10.994 INFO:teuthology.orchestra.run.smithi194.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T16:25:11.028 INFO:teuthology.orchestra.run.smithi194.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T16:25:11.062 INFO:teuthology.orchestra.run.smithi194.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T16:25:11.095 INFO:teuthology.orchestra.run.smithi194.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-25T16:25:11.221 INFO:teuthology.orchestra.run.smithi194.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 998 kB/s | 125 kB 00:00 2024-03-25T16:25:11.796 INFO:teuthology.orchestra.run.smithi194.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 39 kB/s | 22 kB 00:00 2024-03-25T16:25:11.921 INFO:teuthology.orchestra.run.smithi194.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-03-25T16:25:12.021 INFO:teuthology.orchestra.run.smithi194.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 435 kB/s | 43 kB 00:00 2024-03-25T16:25:12.088 INFO:teuthology.orchestra.run.smithi194.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-03-25T16:25:12.180 INFO:teuthology.orchestra.run.smithi194.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.8 MB/s | 358 kB 00:00 2024-03-25T16:25:12.230 INFO:teuthology.orchestra.run.smithi194.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 215 kB/s | 11 kB 00:00 2024-03-25T16:25:12.272 INFO:teuthology.orchestra.run.smithi194.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 423 kB/s | 18 kB 00:00 2024-03-25T16:25:12.323 INFO:teuthology.orchestra.run.smithi194.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 465 kB/s | 23 kB 00:00 2024-03-25T16:25:12.381 INFO:teuthology.orchestra.run.smithi194.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 334 kB/s | 19 kB 00:00 2024-03-25T16:25:12.440 INFO:teuthology.orchestra.run.smithi194.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 337 kB/s | 20 kB 00:00 2024-03-25T16:25:12.490 INFO:teuthology.orchestra.run.smithi194.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 931 kB/s | 46 kB 00:00 2024-03-25T16:25:12.532 INFO:teuthology.orchestra.run.smithi194.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-03-25T16:25:12.632 INFO:teuthology.orchestra.run.smithi194.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-03-25T16:25:12.674 INFO:teuthology.orchestra.run.smithi194.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 395 kB/s | 16 kB 00:00 2024-03-25T16:25:12.725 INFO:teuthology.orchestra.run.smithi194.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-03-25T16:25:12.767 INFO:teuthology.orchestra.run.smithi194.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 851 kB/s | 36 kB 00:00 2024-03-25T16:25:12.842 INFO:teuthology.orchestra.run.smithi194.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-25T16:25:12.942 INFO:teuthology.orchestra.run.smithi194.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-03-25T16:25:12.984 INFO:teuthology.orchestra.run.smithi194.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 479 kB/s | 20 kB 00:00 2024-03-25T16:25:16.659 INFO:teuthology.orchestra.run.smithi194.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 361 kB/s | 4.7 MB 00:13 2024-03-25T16:25:20.737 INFO:teuthology.orchestra.run.smithi156.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 483 kB/s | 17 MB 00:35 2024-03-25T16:25:20.742 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:20.742 INFO:teuthology.orchestra.run.smithi156.stdout:Total 717 kB/s | 30 MB 00:42 2024-03-25T16:25:20.764 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T16:25:20.806 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T16:25:20.806 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T16:25:21.202 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T16:25:21.202 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T16:25:22.525 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T16:25:22.590 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:25:22.629 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:25:22.659 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:25:22.730 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:25:22.774 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:25:22.807 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:25:22.899 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:25:22.977 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:25:23.018 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:25:23.090 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:25:23.135 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:25:23.174 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:25:23.223 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:25:23.278 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:25:23.463 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:25:23.512 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:25:23.550 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:25:23.584 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:25:23.615 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:25:23.691 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:25:23.725 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:25:23.758 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:25:23.799 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:25:23.834 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:25:23.869 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:25:23.958 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:25:23.999 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:25:24.072 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:25:24.207 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:25:24.325 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:25:24.484 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:25:24.513 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:25:24.538 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:25:24.538 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:25:24.538 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:25:24.538 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:25:24.538 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:25:24.539 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:25:25.989 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:25:26.017 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:25:26.017 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:25:26.017 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:25:26.017 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:25:26.017 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:25:26.017 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:25:26.482 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:25:26.507 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:25:26.507 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:25:26.507 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:25:26.507 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:25:26.507 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:25:26.507 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:25:26.733 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:25:26.762 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:25:26.762 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:25:26.762 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:25:26.762 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:25:26.762 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:25:26.763 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:25:26.831 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:25:28.556 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:25:28.556 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:25:28.556 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:25:28.557 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:25:28.557 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:25:28.557 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:25:28.557 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:25:28.557 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:25:28.557 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:25:28.557 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:25:28.557 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:25:28.557 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:25:28.557 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:25:28.557 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:25:28.557 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:25:28.557 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:25:28.557 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:25:28.557 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:25:28.557 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:25:28.558 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:25:28.558 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:25:28.558 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:25:28.558 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:25:28.558 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:25:28.558 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:25:28.558 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:25:28.558 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:25:28.558 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:25:28.558 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:25:28.558 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:25:28.558 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:25:28.558 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:25:28.559 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:25:28.559 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:25:28.559 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:25:28.559 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:25:29.648 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:25:29.648 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:25:29.648 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T16:25:29.648 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:29.648 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:29.648 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:29.648 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:25:29.648 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:29.649 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:29.649 INFO:teuthology.orchestra.run.smithi156.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:25:29.649 INFO:teuthology.orchestra.run.smithi156.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:25:29.649 INFO:teuthology.orchestra.run.smithi156.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:25:29.649 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:25:29.649 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:25:29.649 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:25:29.649 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:25:29.649 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:25:29.649 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:25:29.649 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:25:29.649 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:25:29.650 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:25:29.650 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:25:29.650 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:25:29.650 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:25:29.650 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:25:29.650 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:25:29.650 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:25:29.650 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:25:29.650 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:25:29.650 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:25:29.650 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:25:29.650 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:25:29.650 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:25:29.651 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:25:29.651 INFO:teuthology.orchestra.run.smithi156.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:25:29.651 INFO:teuthology.orchestra.run.smithi156.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:25:29.651 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:25:29.651 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:25:29.651 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:25:29.651 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:25:29.651 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:25:29.996 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-base 2024-03-25T16:25:30.585 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:51 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:25:30.683 INFO:teuthology.orchestra.run.smithi156.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:25:30.736 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:25:30.737 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2024-03-25T16:25:30.737 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:25:30.798 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install cephadm 2024-03-25T16:25:31.362 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:52 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:25:31.513 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:25:31.514 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:25:31.514 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-03-25T16:25:31.514 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:25:31.514 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T16:25:31.514 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:25:31.514 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:25:31.514 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T16:25:31.514 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:25:31.514 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-03-25T16:25:31.514 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:25:31.515 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 747 k 2024-03-25T16:25:31.515 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 754 k 2024-03-25T16:25:31.515 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T16:25:31.999 INFO:teuthology.orchestra.run.smithi194.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 674 kB/s | 17 MB 00:25 2024-03-25T16:25:32.002 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:32.002 INFO:teuthology.orchestra.run.smithi194.stdout:Total 1.0 MB/s | 30 MB 00:29 2024-03-25T16:25:32.027 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T16:25:32.071 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T16:25:32.071 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T16:25:32.490 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T16:25:32.491 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T16:25:33.246 INFO:teuthology.orchestra.run.smithi156.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 432 kB/s | 747 kB 00:01 2024-03-25T16:25:33.246 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:33.247 INFO:teuthology.orchestra.run.smithi156.stdout:Total 431 kB/s | 747 kB 00:01 2024-03-25T16:25:33.247 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T16:25:33.253 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T16:25:33.253 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T16:25:33.259 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T16:25:33.259 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T16:25:33.332 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T16:25:33.756 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:25:33.858 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:25:33.907 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T16:25:33.959 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:25:33.998 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:25:34.037 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:25:34.106 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:25:34.152 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:25:34.185 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:25:34.212 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:25:34.276 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:25:34.354 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:25:34.395 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:25:34.468 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:25:34.488 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:25:34.488 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:25:34.488 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T16:25:34.488 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:25:34.489 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:25:34.489 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:25:34.522 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:25:34.724 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:25:34.724 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:25:34.724 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:25:34.725 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:25:34.857 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:25:34.905 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:25:34.944 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:25:34.978 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:25:35.008 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:25:35.084 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:25:35.118 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:25:35.152 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:25:35.200 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:25:35.236 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:25:35.270 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:25:35.297 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:56 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:25:35.359 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:25:35.393 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:25:35.450 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:25:35.451 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:25:35.451 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-03-25T16:25:35.452 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:25:35.452 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T16:25:35.452 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:25:35.452 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:25:35.452 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T16:25:35.452 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:25:35.452 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-03-25T16:25:35.452 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:25:35.452 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 145 k 2024-03-25T16:25:35.453 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 436 k 2024-03-25T16:25:35.453 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T16:25:35.464 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:25:35.601 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:25:35.728 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:25:35.893 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:25:35.922 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:25:35.947 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:25:35.947 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:25:35.947 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:25:35.948 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:25:35.948 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:25:35.948 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:25:36.119 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 218 kB/s | 145 kB 00:00 2024-03-25T16:25:36.120 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:36.120 INFO:teuthology.orchestra.run.smithi156.stdout:Total 218 kB/s | 145 kB 00:00 2024-03-25T16:25:36.120 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T16:25:36.127 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T16:25:36.128 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T16:25:36.180 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T16:25:36.180 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T16:25:36.320 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T16:25:36.416 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:25:36.441 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:25:36.441 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:25:36.441 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:25:36.441 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:25:37.390 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:25:37.416 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:25:37.416 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:25:37.416 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:25:37.416 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:25:37.416 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:25:37.416 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:25:37.480 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:25:37.481 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:25:37.481 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T16:25:37.481 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:37.481 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:25:37.481 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:25:37.710 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-mgr 2024-03-25T16:25:37.857 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:25:37.880 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:25:37.881 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:25:37.881 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:25:37.881 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:25:37.881 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:25:37.881 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:25:38.091 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:25:38.115 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:25:38.115 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:25:38.115 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:25:38.115 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:25:38.115 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:25:38.115 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:25:38.180 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:25:38.285 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:59 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:25:38.384 INFO:teuthology.orchestra.run.smithi156.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:25:38.436 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:25:38.437 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2024-03-25T16:25:38.437 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:25:38.502 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:25:39.066 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:03:00 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:25:39.215 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:25:39.216 INFO:teuthology.orchestra.run.smithi156.stdout:====================================================================================== 2024-03-25T16:25:39.217 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-03-25T16:25:39.217 INFO:teuthology.orchestra.run.smithi156.stdout:====================================================================================== 2024-03-25T16:25:39.217 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T16:25:39.217 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:25:39.217 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-03-25T16:25:39.217 INFO:teuthology.orchestra.run.smithi156.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:25:39.217 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:25:39.217 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:25:39.217 INFO:teuthology.orchestra.run.smithi156.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:25:39.217 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:25:39.217 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:25:39.217 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:25:39.217 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:25:39.218 INFO:teuthology.orchestra.run.smithi156.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:25:39.218 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:25:39.218 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:25:39.218 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:25:39.218 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T16:25:39.218 INFO:teuthology.orchestra.run.smithi156.stdout:====================================================================================== 2024-03-25T16:25:39.218 INFO:teuthology.orchestra.run.smithi156.stdout:Install 12 Packages 2024-03-25T16:25:39.218 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:25:39.223 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 6.9 M 2024-03-25T16:25:39.223 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 26 M 2024-03-25T16:25:39.223 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T16:25:39.787 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:25:39.787 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:25:39.788 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:25:39.788 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:25:39.788 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:25:39.788 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:25:39.788 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:25:39.788 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:25:39.788 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:25:39.788 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:25:39.788 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:25:39.788 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:25:39.788 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:25:39.788 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:25:39.788 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:25:39.789 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:25:39.789 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:25:39.789 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:25:39.789 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:25:39.789 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:25:39.789 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:25:39.789 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:25:39.789 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:25:39.789 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:25:39.789 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:25:39.789 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:25:39.789 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:25:39.789 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:25:39.789 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:25:39.789 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:25:39.790 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:25:39.790 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:25:39.790 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:25:39.790 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:25:39.790 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:25:39.790 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:25:40.248 INFO:teuthology.orchestra.run.smithi156.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 59 kB/s | 31 kB 00:00 2024-03-25T16:25:40.290 INFO:teuthology.orchestra.run.smithi156.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 26 kB/s | 15 kB 00:00 2024-03-25T16:25:40.432 INFO:teuthology.orchestra.run.smithi156.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 238 kB 00:00 2024-03-25T16:25:40.466 INFO:teuthology.orchestra.run.smithi156.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-03-25T16:25:40.583 INFO:teuthology.orchestra.run.smithi156.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 5.6 MB/s | 863 kB 00:00 2024-03-25T16:25:40.708 INFO:teuthology.orchestra.run.smithi156.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 155 kB/s | 19 kB 00:00 2024-03-25T16:25:40.733 INFO:teuthology.orchestra.run.smithi156.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.0 MB/s | 548 kB 00:00 2024-03-25T16:25:40.792 INFO:teuthology.orchestra.run.smithi156.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.4 MB/s | 144 kB 00:00 2024-03-25T16:25:40.855 INFO:teuthology.orchestra.run.smithi156.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 527 kB/s | 31 kB 00:00 2024-03-25T16:25:40.918 INFO:teuthology.orchestra.run.smithi156.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-25T16:25:40.939 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:25:40.939 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:25:40.939 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T16:25:40.939 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:40.939 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:40.939 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:40.940 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:25:40.940 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:40.940 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:40.940 INFO:teuthology.orchestra.run.smithi194.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:25:40.940 INFO:teuthology.orchestra.run.smithi194.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:25:40.940 INFO:teuthology.orchestra.run.smithi194.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:25:40.940 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:25:40.940 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:25:40.940 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:25:40.940 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:25:40.940 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:25:40.940 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:25:40.941 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:25:40.941 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:25:40.941 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:25:40.941 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:25:40.941 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:25:40.941 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:25:40.941 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:25:40.941 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:25:40.941 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:25:40.941 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:25:40.941 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:25:40.941 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:25:40.942 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:25:40.942 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:25:40.942 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:25:40.942 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:25:40.942 INFO:teuthology.orchestra.run.smithi194.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:25:40.942 INFO:teuthology.orchestra.run.smithi194.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:25:40.942 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:25:40.942 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:25:40.942 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:25:40.942 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:25:40.942 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:25:41.026 INFO:teuthology.orchestra.run.smithi156.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 6.4 MB/s | 2.0 MB 00:00 2024-03-25T16:25:41.321 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-base 2024-03-25T16:25:41.891 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:03:13 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:25:41.990 INFO:teuthology.orchestra.run.smithi194.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:25:42.043 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:25:42.044 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-25T16:25:42.044 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:25:42.123 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install cephadm 2024-03-25T16:25:42.681 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:03:14 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:25:42.831 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:25:42.831 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:25:42.831 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-25T16:25:42.831 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:25:42.831 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T16:25:42.831 INFO:teuthology.orchestra.run.smithi194.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:25:42.831 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:25:42.831 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T16:25:42.832 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:25:42.832 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-25T16:25:42.832 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:25:42.832 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 747 k 2024-03-25T16:25:42.832 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 754 k 2024-03-25T16:25:42.832 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T16:25:44.644 INFO:teuthology.orchestra.run.smithi194.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 413 kB/s | 747 kB 00:01 2024-03-25T16:25:44.645 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:44.645 INFO:teuthology.orchestra.run.smithi194.stdout:Total 412 kB/s | 747 kB 00:01 2024-03-25T16:25:44.646 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T16:25:44.652 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T16:25:44.652 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T16:25:44.658 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T16:25:44.658 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T16:25:44.738 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T16:25:45.131 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:25:45.209 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:25:45.619 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:25:45.759 INFO:teuthology.orchestra.run.smithi156.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 302 kB/s | 1.8 MB 00:06 2024-03-25T16:25:45.763 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:45.764 INFO:teuthology.orchestra.run.smithi156.stdout:Total 1.0 MB/s | 6.9 MB 00:06 2024-03-25T16:25:45.783 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T16:25:45.803 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T16:25:45.803 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T16:25:45.911 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:25:45.911 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:25:45.911 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T16:25:45.911 INFO:teuthology.orchestra.run.smithi194.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:25:45.912 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:25:45.912 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:25:45.969 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T16:25:45.969 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T16:25:46.139 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:25:46.493 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T16:25:46.568 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:25:46.604 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:25:46.651 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:25:46.689 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:25:46.717 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:03:18 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:25:46.793 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:25:46.867 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:25:46.868 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:25:46.869 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-25T16:25:46.869 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:25:46.869 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T16:25:46.869 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:25:46.869 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:25:46.869 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T16:25:46.869 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:25:46.869 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-25T16:25:46.869 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:25:46.869 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 145 k 2024-03-25T16:25:46.869 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 436 k 2024-03-25T16:25:46.869 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T16:25:46.916 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:25:47.001 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:25:47.044 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:25:47.081 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:25:47.112 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:25:47.354 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:25:47.413 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:25:47.849 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 148 kB/s | 145 kB 00:00 2024-03-25T16:25:47.850 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:47.850 INFO:teuthology.orchestra.run.smithi194.stdout:Total 148 kB/s | 145 kB 00:00 2024-03-25T16:25:47.850 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T16:25:47.859 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T16:25:47.859 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T16:25:47.913 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T16:25:47.913 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T16:25:48.051 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T16:25:48.117 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:25:48.141 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:25:48.141 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:25:48.141 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:25:48.141 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:25:48.219 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:25:48.219 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:25:48.219 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:25:48.219 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:25:48.219 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:25:48.219 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:25:48.219 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:25:48.219 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:25:48.219 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:25:48.219 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:25:48.220 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:25:48.220 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:25:48.561 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:25:48.561 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:25:48.561 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T16:25:48.561 INFO:teuthology.orchestra.run.smithi156.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:25:48.562 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:25:48.562 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:25:48.562 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:25:48.562 INFO:teuthology.orchestra.run.smithi156.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:25:48.562 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:25:48.562 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:25:48.562 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:25:48.562 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:25:48.562 INFO:teuthology.orchestra.run.smithi156.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:25:48.562 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:25:48.562 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:25:48.562 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:25:48.562 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:25:48.831 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:25:49.105 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:25:49.105 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:25:49.105 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T16:25:49.105 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:49.105 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:25:49.106 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:25:49.361 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr 2024-03-25T16:25:49.388 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:03:10 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:25:49.540 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:25:49.541 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================ 2024-03-25T16:25:49.541 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-03-25T16:25:49.541 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================ 2024-03-25T16:25:49.541 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T16:25:49.541 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:25:49.541 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-03-25T16:25:49.541 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:25:49.541 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:25:49.542 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:25:49.542 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:25:49.542 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:25:49.542 INFO:teuthology.orchestra.run.smithi156.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:25:49.542 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:25:49.542 INFO:teuthology.orchestra.run.smithi156.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:25:49.542 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:25:49.542 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:25:49.542 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:25:49.542 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:25:49.542 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T16:25:49.542 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================ 2024-03-25T16:25:49.542 INFO:teuthology.orchestra.run.smithi156.stdout:Install 12 Packages 2024-03-25T16:25:49.542 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:25:49.543 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 37 M 2024-03-25T16:25:49.543 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 209 M 2024-03-25T16:25:49.543 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T16:25:49.941 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:03:21 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:25:50.040 INFO:teuthology.orchestra.run.smithi194.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:25:50.041 INFO:teuthology.orchestra.run.smithi156.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 218 kB/s | 32 kB 00:00 2024-03-25T16:25:50.066 INFO:teuthology.orchestra.run.smithi156.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 690 kB/s | 17 kB 00:00 2024-03-25T16:25:50.093 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:25:50.094 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-25T16:25:50.094 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:25:50.100 INFO:teuthology.orchestra.run.smithi156.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T16:25:50.172 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:25:50.258 INFO:teuthology.orchestra.run.smithi156.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 30 MB/s | 4.7 MB 00:00 2024-03-25T16:25:50.392 INFO:teuthology.orchestra.run.smithi156.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-25T16:25:50.443 INFO:teuthology.orchestra.run.smithi156.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.3 MB/s | 217 kB 00:00 2024-03-25T16:25:50.493 INFO:teuthology.orchestra.run.smithi156.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.0 MB/s | 3.0 MB 00:00 2024-03-25T16:25:50.736 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:03:22 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:25:50.793 INFO:teuthology.orchestra.run.smithi156.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-03-25T16:25:50.835 INFO:teuthology.orchestra.run.smithi156.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 803 kB 00:00 2024-03-25T16:25:50.886 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:25:50.887 INFO:teuthology.orchestra.run.smithi194.stdout:====================================================================================== 2024-03-25T16:25:50.887 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-25T16:25:50.887 INFO:teuthology.orchestra.run.smithi194.stdout:====================================================================================== 2024-03-25T16:25:50.887 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T16:25:50.888 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:25:50.888 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-25T16:25:50.888 INFO:teuthology.orchestra.run.smithi194.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:25:50.888 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:25:50.888 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:25:50.888 INFO:teuthology.orchestra.run.smithi194.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:25:50.888 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:25:50.888 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:25:50.888 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:25:50.888 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:25:50.888 INFO:teuthology.orchestra.run.smithi194.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:25:50.888 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:25:50.888 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:25:50.888 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:25:50.889 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T16:25:50.889 INFO:teuthology.orchestra.run.smithi194.stdout:====================================================================================== 2024-03-25T16:25:50.889 INFO:teuthology.orchestra.run.smithi194.stdout:Install 12 Packages 2024-03-25T16:25:50.889 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:25:50.889 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 6.9 M 2024-03-25T16:25:50.889 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 26 M 2024-03-25T16:25:50.889 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T16:25:50.961 INFO:teuthology.orchestra.run.smithi156.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-25T16:25:50.986 INFO:teuthology.orchestra.run.smithi156.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-03-25T16:25:51.427 INFO:teuthology.orchestra.run.smithi194.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 70 kB/s | 31 kB 00:00 2024-03-25T16:25:51.569 INFO:teuthology.orchestra.run.smithi194.stdout:(2/12): protobuf-3.14.0-13.el9.x86_64.rpm 7.1 MB/s | 1.0 MB 00:00 2024-03-25T16:25:51.611 INFO:teuthology.orchestra.run.smithi194.stdout:(3/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 24 kB/s | 15 kB 00:00 2024-03-25T16:25:51.636 INFO:teuthology.orchestra.run.smithi194.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 3.5 MB/s | 238 kB 00:00 2024-03-25T16:25:51.928 INFO:teuthology.orchestra.run.smithi194.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.8 MB/s | 548 kB 00:00 2024-03-25T16:25:52.037 INFO:teuthology.orchestra.run.smithi194.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 180 kB/s | 19 kB 00:00 2024-03-25T16:25:52.395 INFO:teuthology.orchestra.run.smithi194.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.1 MB/s | 863 kB 00:00 2024-03-25T16:25:52.471 INFO:teuthology.orchestra.run.smithi194.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 4.7 MB/s | 2.0 MB 00:00 2024-03-25T16:25:52.571 INFO:teuthology.orchestra.run.smithi194.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 825 kB/s | 144 kB 00:00 2024-03-25T16:25:52.596 INFO:teuthology.orchestra.run.smithi194.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 248 kB/s | 31 kB 00:00 2024-03-25T16:25:52.729 INFO:teuthology.orchestra.run.smithi194.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.2 MB/s | 188 kB 00:00 2024-03-25T16:25:53.580 INFO:teuthology.orchestra.run.smithi194.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 701 kB/s | 1.8 MB 00:02 2024-03-25T16:25:53.583 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:53.583 INFO:teuthology.orchestra.run.smithi194.stdout:Total 2.5 MB/s | 6.9 MB 00:02 2024-03-25T16:25:53.603 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T16:25:53.623 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T16:25:53.623 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T16:25:53.808 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T16:25:53.809 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T16:25:54.333 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T16:25:54.414 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:25:54.457 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:25:54.504 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:25:54.551 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:25:54.674 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:25:54.801 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:25:54.907 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:25:54.955 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:25:54.992 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:25:55.023 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:25:55.265 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:25:55.307 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:25:56.129 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:25:56.129 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:25:56.129 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:25:56.129 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:25:56.129 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:25:56.129 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:25:56.129 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:25:56.129 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:25:56.129 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:25:56.129 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:25:56.130 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:25:56.130 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:25:56.468 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:25:56.468 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:25:56.468 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T16:25:56.469 INFO:teuthology.orchestra.run.smithi194.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:25:56.469 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:25:56.469 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:25:56.469 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:25:56.469 INFO:teuthology.orchestra.run.smithi194.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:25:56.469 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:25:56.469 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:25:56.469 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:25:56.469 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:25:56.469 INFO:teuthology.orchestra.run.smithi194.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:25:56.469 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:25:56.470 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:25:56.470 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:25:56.470 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:25:56.723 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:25:57.283 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:03:29 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:25:57.434 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:25:57.436 INFO:teuthology.orchestra.run.smithi194.stdout:============================================================================================ 2024-03-25T16:25:57.436 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-25T16:25:57.436 INFO:teuthology.orchestra.run.smithi194.stdout:============================================================================================ 2024-03-25T16:25:57.436 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T16:25:57.436 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:25:57.436 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-25T16:25:57.436 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:25:57.436 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:25:57.437 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:25:57.437 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:25:57.437 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:25:57.437 INFO:teuthology.orchestra.run.smithi194.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:25:57.437 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:25:57.437 INFO:teuthology.orchestra.run.smithi194.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:25:57.437 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:25:57.437 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:25:57.437 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:25:57.437 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:25:57.438 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T16:25:57.438 INFO:teuthology.orchestra.run.smithi194.stdout:============================================================================================ 2024-03-25T16:25:57.438 INFO:teuthology.orchestra.run.smithi194.stdout:Install 12 Packages 2024-03-25T16:25:57.438 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:25:57.439 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 37 M 2024-03-25T16:25:57.439 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 209 M 2024-03-25T16:25:57.439 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T16:25:57.877 INFO:teuthology.orchestra.run.smithi194.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 152 kB/s | 32 kB 00:00 2024-03-25T16:25:57.944 INFO:teuthology.orchestra.run.smithi194.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 257 kB/s | 17 kB 00:00 2024-03-25T16:25:58.020 INFO:teuthology.orchestra.run.smithi194.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 456 kB/s | 34 kB 00:00 2024-03-25T16:25:58.253 INFO:teuthology.orchestra.run.smithi194.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.1 MB/s | 3.0 MB 00:00 2024-03-25T16:25:58.545 INFO:teuthology.orchestra.run.smithi194.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 9.0 MB/s | 4.7 MB 00:00 2024-03-25T16:25:58.620 INFO:teuthology.orchestra.run.smithi194.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 4.8 MB 00:00 2024-03-25T16:25:58.645 INFO:teuthology.orchestra.run.smithi194.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-25T16:25:59.004 INFO:teuthology.orchestra.run.smithi194.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-03-25T16:25:59.088 INFO:teuthology.orchestra.run.smithi194.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-14 5.2 MB/s | 7.4 MB 00:01 2024-03-25T16:25:59.146 INFO:teuthology.orchestra.run.smithi194.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 803 kB 00:00 2024-03-25T16:25:59.296 INFO:teuthology.orchestra.run.smithi194.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 669 kB/s | 195 kB 00:00 2024-03-25T16:25:59.355 INFO:teuthology.orchestra.run.smithi194.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 919 kB/s | 245 kB 00:00 2024-03-25T16:25:59.357 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:59.357 INFO:teuthology.orchestra.run.smithi194.stdout:Total 19 MB/s | 37 MB 00:01 2024-03-25T16:25:59.399 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T16:25:59.422 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T16:25:59.422 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T16:25:59.786 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T16:25:59.786 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T16:26:00.491 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T16:26:00.561 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:26:00.593 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:26:00.686 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:26:00.729 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:26:00.906 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:26:00.936 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:26:01.013 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:26:01.658 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:26:01.765 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:26:03.017 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:26:04.043 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:26:04.076 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:26:06.557 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:26:06.557 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:26:06.557 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:26:06.557 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:26:06.557 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:26:06.557 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:26:06.557 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:26:06.557 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:26:06.558 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:26:06.558 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:26:06.558 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:26:06.558 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:26:06.634 INFO:teuthology.orchestra.run.smithi156.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 452 kB/s | 7.4 MB 00:16 2024-03-25T16:26:06.636 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:06.637 INFO:teuthology.orchestra.run.smithi156.stdout:Total 2.2 MB/s | 37 MB 00:17 2024-03-25T16:26:06.679 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T16:26:06.705 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T16:26:06.705 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T16:26:06.979 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:26:06.979 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:06.979 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T16:26:06.979 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:26:06.979 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:26:06.979 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:26:06.979 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:26:06.979 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:26:06.979 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:26:06.979 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:26:06.979 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:26:06.980 INFO:teuthology.orchestra.run.smithi194.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:26:06.980 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:26:06.980 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:26:06.980 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:26:06.980 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:06.980 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:26:07.075 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T16:26:07.076 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T16:26:07.363 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:26:07.782 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T16:26:07.857 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:26:07.888 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:26:07.948 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:03:39 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:26:07.980 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:26:08.016 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:26:08.108 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:26:08.109 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================== 2024-03-25T16:26:08.109 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-25T16:26:08.109 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================== 2024-03-25T16:26:08.109 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T16:26:08.109 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:26:08.109 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-25T16:26:08.110 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:26:08.110 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:26:08.110 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:26:08.110 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:26:08.110 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:26:08.110 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:26:08.110 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:26:08.110 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:26:08.110 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:26:08.110 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:26:08.110 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:26:08.110 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:26:08.110 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:08.111 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T16:26:08.111 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================== 2024-03-25T16:26:08.111 INFO:teuthology.orchestra.run.smithi194.stdout:Install 13 Packages 2024-03-25T16:26:08.111 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:08.111 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 2.0 M 2024-03-25T16:26:08.111 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 26 M 2024-03-25T16:26:08.111 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T16:26:08.198 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:26:08.232 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:26:08.267 INFO:teuthology.orchestra.run.smithi194.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 167 kB/s | 18 kB 00:00 2024-03-25T16:26:08.316 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:26:08.317 INFO:teuthology.orchestra.run.smithi194.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 154 kB/s | 24 kB 00:00 2024-03-25T16:26:08.359 INFO:teuthology.orchestra.run.smithi194.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2024-03-25T16:26:08.384 INFO:teuthology.orchestra.run.smithi194.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-25T16:26:08.410 INFO:teuthology.orchestra.run.smithi194.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:26:08.435 INFO:teuthology.orchestra.run.smithi194.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 872 kB/s | 43 kB 00:00 2024-03-25T16:26:08.660 INFO:teuthology.orchestra.run.smithi194.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 67 kB/s | 15 kB 00:00 2024-03-25T16:26:08.694 INFO:teuthology.orchestra.run.smithi194.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 113 kB/s | 32 kB 00:00 2024-03-25T16:26:08.719 INFO:teuthology.orchestra.run.smithi194.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 93 kB/s | 52 kB 00:00 2024-03-25T16:26:08.964 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:26:09.011 INFO:teuthology.orchestra.run.smithi194.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 578 kB/s | 203 kB 00:00 2024-03-25T16:26:09.061 INFO:teuthology.orchestra.run.smithi194.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 173 kB/s | 59 kB 00:00 2024-03-25T16:26:09.077 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:26:09.128 INFO:teuthology.orchestra.run.smithi194.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 770 kB/s | 90 kB 00:00 2024-03-25T16:26:09.212 INFO:teuthology.orchestra.run.smithi194.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.0 MB/s | 1.0 MB 00:00 2024-03-25T16:26:09.213 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:09.214 INFO:teuthology.orchestra.run.smithi194.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-03-25T16:26:09.214 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T16:26:09.227 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T16:26:09.227 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T16:26:09.316 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T16:26:09.318 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T16:26:09.826 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T16:26:10.000 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:26:10.062 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:26:10.118 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:26:10.155 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:26:10.196 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:26:10.289 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:26:10.302 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:26:10.390 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:26:10.447 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:26:11.166 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:26:11.211 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:26:11.245 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:26:11.294 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:26:11.315 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:26:11.327 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:26:11.349 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:26:12.087 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:26:12.087 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:26:12.087 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:26:12.087 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:26:12.087 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:26:12.088 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:26:12.088 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:26:12.088 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:26:12.088 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:26:12.088 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:26:12.088 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:26:12.088 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:26:12.088 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:26:12.435 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:26:12.435 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:12.435 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T16:26:12.435 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:26:12.435 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:26:12.436 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:26:12.436 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:26:12.436 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:26:12.436 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:26:12.436 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:26:12.436 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:26:12.436 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:26:12.436 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:26:12.436 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:26:12.436 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:26:12.436 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:26:12.436 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:12.436 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:26:12.724 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:26:13.288 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:03:45 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:26:13.438 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:26:13.438 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================== 2024-03-25T16:26:13.438 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-25T16:26:13.439 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================== 2024-03-25T16:26:13.439 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T16:26:13.439 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:26:13.439 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-25T16:26:13.439 INFO:teuthology.orchestra.run.smithi194.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:26:13.439 INFO:teuthology.orchestra.run.smithi194.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:26:13.439 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:26:13.439 INFO:teuthology.orchestra.run.smithi194.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:26:13.439 INFO:teuthology.orchestra.run.smithi194.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:26:13.439 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:13.439 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T16:26:13.439 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================== 2024-03-25T16:26:13.440 INFO:teuthology.orchestra.run.smithi194.stdout:Install 6 Packages 2024-03-25T16:26:13.440 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:13.440 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 6.8 M 2024-03-25T16:26:13.440 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 33 M 2024-03-25T16:26:13.440 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T16:26:13.867 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:26:13.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:26:13.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:26:13.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:26:13.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:26:13.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:26:13.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:26:13.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:26:13.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:26:13.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:26:13.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:26:13.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:26:14.071 INFO:teuthology.orchestra.run.smithi194.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-25T16:26:14.113 INFO:teuthology.orchestra.run.smithi194.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 776 kB/s | 248 kB 00:00 2024-03-25T16:26:14.296 INFO:teuthology.orchestra.run.smithi194.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 359 kB/s | 58 kB 00:00 2024-03-25T16:26:14.333 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:26:14.333 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:26:14.333 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T16:26:14.333 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:26:14.333 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:26:14.333 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:26:14.333 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:26:14.334 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:26:14.334 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:26:14.334 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:26:14.334 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:26:14.334 INFO:teuthology.orchestra.run.smithi156.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:26:14.334 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:26:14.334 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:26:14.334 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:26:14.334 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:26:14.334 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:26:14.355 INFO:teuthology.orchestra.run.smithi194.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.9 MB/s | 548 kB 00:00 2024-03-25T16:26:14.380 INFO:teuthology.orchestra.run.smithi194.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 612 kB/s | 51 kB 00:00 2024-03-25T16:26:14.589 INFO:teuthology.orchestra.run.smithi194.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 173 kB/s | 138 kB 00:00 2024-03-25T16:26:14.592 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:14.592 INFO:teuthology.orchestra.run.smithi194.stdout:Total 5.9 MB/s | 6.8 MB 00:01 2024-03-25T16:26:14.605 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T16:26:14.617 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T16:26:14.617 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T16:26:14.699 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:26:14.706 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T16:26:14.708 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T16:26:15.004 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T16:26:15.168 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:26:15.223 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:26:15.271 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:03:36 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:26:15.423 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:26:15.424 INFO:teuthology.orchestra.run.smithi156.stdout:======================================================================================== 2024-03-25T16:26:15.424 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-03-25T16:26:15.424 INFO:teuthology.orchestra.run.smithi156.stdout:======================================================================================== 2024-03-25T16:26:15.424 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T16:26:15.424 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:26:15.424 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-03-25T16:26:15.424 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:26:15.424 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:26:15.424 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:26:15.425 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:26:15.425 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:26:15.425 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:26:15.425 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:26:15.425 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:26:15.425 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:26:15.425 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:26:15.425 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:26:15.425 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:26:15.425 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:26:15.425 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T16:26:15.425 INFO:teuthology.orchestra.run.smithi156.stdout:======================================================================================== 2024-03-25T16:26:15.425 INFO:teuthology.orchestra.run.smithi156.stdout:Install 13 Packages 2024-03-25T16:26:15.425 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:26:15.426 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 2.0 M 2024-03-25T16:26:15.426 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 26 M 2024-03-25T16:26:15.426 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T16:26:15.667 INFO:teuthology.orchestra.run.smithi156.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 252 kB/s | 24 kB 00:00 2024-03-25T16:26:15.709 INFO:teuthology.orchestra.run.smithi156.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 130 kB/s | 18 kB 00:00 2024-03-25T16:26:15.733 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:26:15.751 INFO:teuthology.orchestra.run.smithi156.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.8 MB/s | 174 kB 00:00 2024-03-25T16:26:15.776 INFO:teuthology.orchestra.run.smithi156.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-25T16:26:15.802 INFO:teuthology.orchestra.run.smithi156.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:26:15.808 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:26:15.827 INFO:teuthology.orchestra.run.smithi156.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 869 kB/s | 43 kB 00:00 2024-03-25T16:26:15.878 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:26:15.902 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:26:16.002 INFO:teuthology.orchestra.run.smithi156.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 86 kB/s | 15 kB 00:00 2024-03-25T16:26:16.119 INFO:teuthology.orchestra.run.smithi156.stdout:(8/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 95 kB/s | 52 kB 00:00 2024-03-25T16:26:16.269 INFO:teuthology.orchestra.run.smithi156.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 758 kB/s | 203 kB 00:00 2024-03-25T16:26:16.345 INFO:teuthology.orchestra.run.smithi156.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 789 kB/s | 59 kB 00:00 2024-03-25T16:26:16.428 INFO:teuthology.orchestra.run.smithi156.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.1 MB/s | 90 kB 00:00 2024-03-25T16:26:16.731 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:26:16.732 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:26:16.732 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:26:16.732 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:26:16.732 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:26:16.732 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:26:17.033 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:26:17.034 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:17.034 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T16:26:17.034 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:26:17.034 INFO:teuthology.orchestra.run.smithi194.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:26:17.034 INFO:teuthology.orchestra.run.smithi194.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:26:17.034 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:26:17.034 INFO:teuthology.orchestra.run.smithi194.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:26:17.034 INFO:teuthology.orchestra.run.smithi194.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:26:17.034 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:17.034 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:26:17.284 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-fuse 2024-03-25T16:26:17.849 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:03:49 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:26:17.999 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:26:18.000 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:26:18.000 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-25T16:26:18.000 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:26:18.000 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T16:26:18.000 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:26:18.001 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-25T16:26:18.001 INFO:teuthology.orchestra.run.smithi194.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:26:18.001 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:18.001 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T16:26:18.001 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:26:18.001 INFO:teuthology.orchestra.run.smithi194.stdout:Install 2 Packages 2024-03-25T16:26:18.001 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:18.001 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 964 k 2024-03-25T16:26:18.001 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 2.8 M 2024-03-25T16:26:18.001 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T16:26:18.131 INFO:teuthology.orchestra.run.smithi194.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 606 kB/s | 79 kB 00:00 2024-03-25T16:26:20.248 INFO:teuthology.orchestra.run.smithi194.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 394 kB/s | 885 kB 00:02 2024-03-25T16:26:20.249 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:20.250 INFO:teuthology.orchestra.run.smithi194.stdout:Total 428 kB/s | 964 kB 00:02 2024-03-25T16:26:20.250 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T16:26:20.273 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T16:26:20.273 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T16:26:20.350 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T16:26:20.350 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T16:26:20.535 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T16:26:20.674 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:26:20.747 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:26:21.408 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:26:21.408 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:26:21.697 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:26:21.697 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:21.697 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T16:26:21.698 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:26:21.698 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:21.698 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:26:22.004 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-volume 2024-03-25T16:26:22.576 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:03:54 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:26:22.732 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:26:22.733 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:26:22.733 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-25T16:26:22.733 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:26:22.733 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T16:26:22.733 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:26:22.733 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-25T16:26:22.733 INFO:teuthology.orchestra.run.smithi194.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:26:22.734 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:22.734 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T16:26:22.734 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:26:22.734 INFO:teuthology.orchestra.run.smithi194.stdout:Install 2 Packages 2024-03-25T16:26:22.734 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:22.734 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 332 k 2024-03-25T16:26:22.734 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 1.5 M 2024-03-25T16:26:22.734 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T16:26:22.845 INFO:teuthology.orchestra.run.smithi194.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 622 kB/s | 69 kB 00:00 2024-03-25T16:26:24.020 INFO:teuthology.orchestra.run.smithi194.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 205 kB/s | 264 kB 00:01 2024-03-25T16:26:24.022 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:24.022 INFO:teuthology.orchestra.run.smithi194.stdout:Total 258 kB/s | 332 kB 00:01 2024-03-25T16:26:24.022 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T16:26:24.032 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T16:26:24.032 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T16:26:24.072 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T16:26:24.072 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T16:26:24.225 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T16:26:24.394 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:26:24.431 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:26:24.447 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:26:24.448 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:26:24.448 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:26:24.448 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:25.217 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:26:25.549 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:26:25.549 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:25.549 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T16:26:25.549 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:26:25.549 INFO:teuthology.orchestra.run.smithi194.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:26:25.549 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:25.550 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:26:25.905 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librados-devel 2024-03-25T16:26:26.481 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:03:58 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:26:26.632 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:26:26.632 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:26:26.632 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-25T16:26:26.632 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:26:26.633 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T16:26:26.633 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:26:26.633 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:26.633 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T16:26:26.633 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:26:26.633 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-25T16:26:26.633 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:26.633 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 127 k 2024-03-25T16:26:26.633 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 456 k 2024-03-25T16:26:26.634 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T16:26:27.275 INFO:teuthology.orchestra.run.smithi194.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 197 kB/s | 127 kB 00:00 2024-03-25T16:26:27.276 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:27.276 INFO:teuthology.orchestra.run.smithi194.stdout:Total 197 kB/s | 127 kB 00:00 2024-03-25T16:26:27.276 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T16:26:27.282 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T16:26:27.282 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T16:26:27.338 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T16:26:27.338 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T16:26:27.483 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T16:26:27.582 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:28.039 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:28.320 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:28.321 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:28.321 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T16:26:28.321 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:28.321 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:28.321 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:26:28.608 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install libcephfs2 2024-03-25T16:26:29.188 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:04:01 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:26:29.287 INFO:teuthology.orchestra.run.smithi194.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:26:29.342 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:26:29.343 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-25T16:26:29.344 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:26:29.421 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install libcephfs-devel 2024-03-25T16:26:29.988 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:04:01 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:26:30.138 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:26:30.138 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:26:30.139 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-25T16:26:30.139 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:26:30.139 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T16:26:30.139 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:26:30.139 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:30.139 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T16:26:30.139 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:26:30.139 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-25T16:26:30.139 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:30.139 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 31 k 2024-03-25T16:26:30.139 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 140 k 2024-03-25T16:26:30.139 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T16:26:30.644 INFO:teuthology.orchestra.run.smithi194.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 62 kB/s | 31 kB 00:00 2024-03-25T16:26:30.644 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:30.645 INFO:teuthology.orchestra.run.smithi194.stdout:Total 62 kB/s | 31 kB 00:00 2024-03-25T16:26:30.645 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T16:26:30.648 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T16:26:30.648 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T16:26:30.673 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T16:26:30.674 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T16:26:30.811 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T16:26:30.900 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:31.322 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:31.401 INFO:teuthology.orchestra.run.smithi156.stdout:(12/13): python3-cachetools-4.2.4-1.el9.noarch. 2.1 kB/s | 32 kB 00:15 2024-03-25T16:26:31.580 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:31.580 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:31.580 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T16:26:31.580 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:31.580 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:31.580 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:26:31.747 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librados2 2024-03-25T16:26:32.093 INFO:teuthology.orchestra.run.smithi156.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 65 kB/s | 1.0 MB 00:15 2024-03-25T16:26:32.096 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:32.096 INFO:teuthology.orchestra.run.smithi156.stdout:Total 126 kB/s | 2.0 MB 00:16 2024-03-25T16:26:32.096 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T16:26:32.111 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T16:26:32.111 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T16:26:32.212 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T16:26:32.213 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T16:26:32.314 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:04:04 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:26:32.412 INFO:teuthology.orchestra.run.smithi194.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:26:32.463 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:26:32.464 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-25T16:26:32.464 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:26:32.521 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librbd1 2024-03-25T16:26:32.741 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T16:26:32.908 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:26:32.960 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:26:33.025 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:26:33.062 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:26:33.094 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:04:05 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:26:33.103 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:26:33.192 INFO:teuthology.orchestra.run.smithi194.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:26:33.197 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:26:33.244 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:26:33.245 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-25T16:26:33.245 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:26:33.300 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rados 2024-03-25T16:26:33.325 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:26:33.371 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:26:33.864 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:04:05 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:26:33.960 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:26:34.013 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:26:34.014 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-25T16:26:34.014 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:26:34.069 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rgw 2024-03-25T16:26:34.100 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:26:34.153 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:26:34.186 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:26:34.235 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:26:34.264 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:26:34.638 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:04:06 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:26:34.735 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:26:34.788 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:26:34.789 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-25T16:26:34.789 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:26:34.844 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-cephfs 2024-03-25T16:26:35.004 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:26:35.004 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:26:35.005 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:26:35.005 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:26:35.005 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:26:35.005 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:26:35.005 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:26:35.005 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:26:35.005 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:26:35.005 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:26:35.005 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:26:35.005 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:26:35.005 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:26:35.379 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:26:35.379 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:26:35.379 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T16:26:35.379 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:26:35.379 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:26:35.380 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:26:35.380 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:26:35.380 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:26:35.380 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:26:35.380 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:26:35.380 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:26:35.380 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:26:35.380 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:26:35.380 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:26:35.380 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:26:35.380 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:26:35.380 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:26:35.380 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:26:35.412 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:04:07 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:26:35.508 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:26:35.559 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:26:35.560 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-25T16:26:35.560 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:26:35.616 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rbd 2024-03-25T16:26:35.632 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:26:36.184 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:04:08 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:26:36.200 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:03:57 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:26:36.282 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:26:36.334 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:26:36.335 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-25T16:26:36.335 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:26:36.354 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:26:36.355 INFO:teuthology.orchestra.run.smithi156.stdout:======================================================================================== 2024-03-25T16:26:36.355 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-03-25T16:26:36.355 INFO:teuthology.orchestra.run.smithi156.stdout:======================================================================================== 2024-03-25T16:26:36.355 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T16:26:36.355 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:26:36.355 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-03-25T16:26:36.356 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:26:36.356 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:26:36.356 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:26:36.356 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:26:36.356 INFO:teuthology.orchestra.run.smithi156.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:26:36.356 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:26:36.356 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T16:26:36.356 INFO:teuthology.orchestra.run.smithi156.stdout:======================================================================================== 2024-03-25T16:26:36.356 INFO:teuthology.orchestra.run.smithi156.stdout:Install 6 Packages 2024-03-25T16:26:36.356 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:26:36.357 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 6.8 M 2024-03-25T16:26:36.357 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 33 M 2024-03-25T16:26:36.357 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T16:26:36.394 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-fuse 2024-03-25T16:26:36.964 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:04:08 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:26:37.034 INFO:teuthology.orchestra.run.smithi156.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-25T16:26:37.104 INFO:teuthology.orchestra.run.smithi156.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 626 kB/s | 248 kB 00:00 2024-03-25T16:26:37.118 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:26:37.118 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:26:37.119 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-03-25T16:26:37.119 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:26:37.119 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T16:26:37.119 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:26:37.119 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:37.119 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T16:26:37.119 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:26:37.119 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-25T16:26:37.119 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:37.119 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 85 k 2024-03-25T16:26:37.119 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 231 k 2024-03-25T16:26:37.119 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T16:26:37.155 INFO:teuthology.orchestra.run.smithi156.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.5 MB/s | 548 kB 00:00 2024-03-25T16:26:37.180 INFO:teuthology.orchestra.run.smithi156.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 765 kB/s | 58 kB 00:00 2024-03-25T16:26:37.205 INFO:teuthology.orchestra.run.smithi156.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-03-25T16:26:37.655 INFO:teuthology.orchestra.run.smithi156.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 146 kB/s | 138 kB 00:00 2024-03-25T16:26:37.656 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:37.657 INFO:teuthology.orchestra.run.smithi156.stdout:Total 5.2 MB/s | 6.8 MB 00:01 2024-03-25T16:26:37.670 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T16:26:37.681 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T16:26:37.681 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T16:26:37.730 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 140 kB/s | 85 kB 00:00 2024-03-25T16:26:37.731 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:37.731 INFO:teuthology.orchestra.run.smithi194.stdout:Total 139 kB/s | 85 kB 00:00 2024-03-25T16:26:37.731 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T16:26:37.738 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T16:26:37.738 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T16:26:37.772 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T16:26:37.772 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T16:26:37.794 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T16:26:37.794 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T16:26:37.930 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T16:26:38.046 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:38.071 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T16:26:38.229 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:26:38.273 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:26:38.467 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:38.785 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:38.785 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:38.785 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T16:26:38.785 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:38.785 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:38.785 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:26:38.802 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:26:38.885 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:26:38.946 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:26:38.969 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:26:39.012 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-mirror 2024-03-25T16:26:39.609 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:04:11 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:26:39.764 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:26:39.765 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:26:39.765 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-25T16:26:39.765 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:26:39.766 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T16:26:39.766 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:26:39.766 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:39.766 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T16:26:39.766 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:26:39.766 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-25T16:26:39.766 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:39.766 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 3.1 M 2024-03-25T16:26:39.766 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 13 M 2024-03-25T16:26:39.766 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T16:26:39.828 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:26:39.828 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:26:39.828 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:26:39.828 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:26:39.828 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:26:39.829 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:26:40.121 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:26:40.121 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:26:40.121 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T16:26:40.121 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:26:40.121 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:26:40.121 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:26:40.121 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:26:40.121 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:26:40.121 INFO:teuthology.orchestra.run.smithi156.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:26:40.121 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:26:40.122 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:26:40.356 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-fuse 2024-03-25T16:26:40.928 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:04:01 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:26:41.079 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:26:41.080 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:26:41.080 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-03-25T16:26:41.080 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:26:41.080 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T16:26:41.080 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:26:41.080 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-03-25T16:26:41.080 INFO:teuthology.orchestra.run.smithi156.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:26:41.081 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:26:41.081 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T16:26:41.081 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:26:41.081 INFO:teuthology.orchestra.run.smithi156.stdout:Install 2 Packages 2024-03-25T16:26:41.081 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:26:41.081 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 964 k 2024-03-25T16:26:41.081 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 2.8 M 2024-03-25T16:26:41.081 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T16:26:41.207 INFO:teuthology.orchestra.run.smithi156.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 625 kB/s | 79 kB 00:00 2024-03-25T16:26:46.641 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 460 kB/s | 3.1 MB 00:06 2024-03-25T16:26:46.676 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:46.676 INFO:teuthology.orchestra.run.smithi194.stdout:Total 460 kB/s | 3.1 MB 00:06 2024-03-25T16:26:46.676 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T16:26:46.676 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T16:26:46.676 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T16:26:46.730 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T16:26:46.730 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T16:26:46.948 INFO:teuthology.orchestra.run.smithi156.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 151 kB/s | 885 kB 00:05 2024-03-25T16:26:46.950 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:46.950 INFO:teuthology.orchestra.run.smithi156.stdout:Total 164 kB/s | 964 kB 00:05 2024-03-25T16:26:46.951 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T16:26:46.974 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T16:26:46.975 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T16:26:47.051 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T16:26:47.052 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T16:26:47.159 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T16:26:47.217 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:47.233 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T16:26:47.241 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:47.241 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:26:47.241 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:26:47.242 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:26:47.242 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:26:47.242 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:47.382 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:26:47.447 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:26:48.170 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:26:48.170 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:26:48.369 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:48.369 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:48.369 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T16:26:48.369 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:48.369 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:48.369 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:26:48.486 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:26:48.645 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:26:48.645 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T16:26:48.645 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:26:48.645 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:26:48.645 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:26:48.646 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-nbd 2024-03-25T16:26:48.776 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-volume 2024-03-25T16:26:49.235 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:04:21 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:26:49.363 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:04:10 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:26:49.401 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:26:49.402 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:26:49.402 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-03-25T16:26:49.402 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:26:49.402 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T16:26:49.402 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:26:49.402 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:49.402 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T16:26:49.403 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:26:49.403 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-25T16:26:49.403 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:49.403 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 171 k 2024-03-25T16:26:49.403 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 494 k 2024-03-25T16:26:49.403 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T16:26:49.521 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:26:49.523 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:26:49.524 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-03-25T16:26:49.524 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:26:49.524 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T16:26:49.524 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:26:49.524 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-03-25T16:26:49.524 INFO:teuthology.orchestra.run.smithi156.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:26:49.524 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:26:49.524 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T16:26:49.524 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:26:49.524 INFO:teuthology.orchestra.run.smithi156.stdout:Install 2 Packages 2024-03-25T16:26:49.524 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:26:49.524 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 332 k 2024-03-25T16:26:49.524 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 1.5 M 2024-03-25T16:26:49.525 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T16:26:49.831 INFO:teuthology.orchestra.run.smithi156.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 224 kB/s | 69 kB 00:00 2024-03-25T16:26:50.111 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 243 kB/s | 171 kB 00:00 2024-03-25T16:26:50.112 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:50.112 INFO:teuthology.orchestra.run.smithi194.stdout:Total 242 kB/s | 171 kB 00:00 2024-03-25T16:26:50.112 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T16:26:50.120 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T16:26:50.120 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T16:26:50.176 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T16:26:50.176 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T16:26:50.324 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T16:26:50.410 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:50.881 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:51.168 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:51.169 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:51.169 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T16:26:51.169 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:51.169 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:51.169 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:26:51.223 INFO:teuthology.orchestra.run.smithi156.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 155 kB/s | 264 kB 00:01 2024-03-25T16:26:51.225 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:51.225 INFO:teuthology.orchestra.run.smithi156.stdout:Total 195 kB/s | 332 kB 00:01 2024-03-25T16:26:51.225 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T16:26:51.234 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T16:26:51.234 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T16:26:51.275 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T16:26:51.275 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T16:26:51.413 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:26:51.422 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T16:26:51.607 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:26:51.642 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:26:51.658 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:26:51.659 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:26:51.659 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:26:51.659 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:26:51.989 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:04:23 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:26:52.143 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:26:52.144 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:26:52.144 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-25T16:26:52.144 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:26:52.144 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T16:26:52.144 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:26:52.144 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-25T16:26:52.144 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:26:52.144 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:52.144 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T16:26:52.144 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:26:52.144 INFO:teuthology.orchestra.run.smithi194.stdout:Install 2 Packages 2024-03-25T16:26:52.144 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:26:52.145 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 91 M 2024-03-25T16:26:52.145 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 498 M 2024-03-25T16:26:52.145 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T16:26:52.425 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:26:52.784 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:26:52.784 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:26:52.784 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T16:26:52.785 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:26:52.785 INFO:teuthology.orchestra.run.smithi156.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:26:52.785 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:26:52.785 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:26:53.122 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install librados-devel 2024-03-25T16:26:53.702 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:04:14 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:26:53.852 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:26:53.852 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:26:53.852 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-03-25T16:26:53.853 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:26:53.853 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T16:26:53.853 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:26:53.853 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:26:53.853 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T16:26:53.853 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:26:53.853 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-03-25T16:26:53.853 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:26:53.854 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 127 k 2024-03-25T16:26:53.854 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 456 k 2024-03-25T16:26:53.854 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T16:26:54.479 INFO:teuthology.orchestra.run.smithi156.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 202 kB/s | 127 kB 00:00 2024-03-25T16:26:54.480 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:54.480 INFO:teuthology.orchestra.run.smithi156.stdout:Total 202 kB/s | 127 kB 00:00 2024-03-25T16:26:54.480 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T16:26:54.486 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T16:26:54.486 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T16:26:54.542 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T16:26:54.542 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T16:26:54.682 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T16:26:54.787 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:55.252 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:55.518 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:55.518 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:26:55.518 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T16:26:55.518 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:55.518 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:26:55.519 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:26:55.747 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install libcephfs2 2024-03-25T16:26:56.324 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:04:17 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:26:56.422 INFO:teuthology.orchestra.run.smithi156.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:26:56.478 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:26:56.478 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2024-03-25T16:26:56.478 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:26:56.542 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install libcephfs-devel 2024-03-25T16:26:57.118 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:04:18 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:26:57.269 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:26:57.270 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:26:57.270 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-03-25T16:26:57.270 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:26:57.270 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T16:26:57.270 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:26:57.270 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:26:57.270 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T16:26:57.271 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:26:57.271 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-03-25T16:26:57.271 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:26:57.271 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 31 k 2024-03-25T16:26:57.271 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 140 k 2024-03-25T16:26:57.271 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T16:26:57.566 INFO:teuthology.orchestra.run.smithi156.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 105 kB/s | 31 kB 00:00 2024-03-25T16:26:57.567 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:57.568 INFO:teuthology.orchestra.run.smithi156.stdout:Total 105 kB/s | 31 kB 00:00 2024-03-25T16:26:57.568 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T16:26:57.571 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T16:26:57.571 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T16:26:57.604 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T16:26:57.605 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T16:26:57.725 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T16:26:57.840 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:58.226 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:58.482 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:58.482 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:26:58.482 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T16:26:58.482 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:58.482 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:26:58.483 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:26:58.632 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install librados2 2024-03-25T16:26:59.197 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:04:20 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:26:59.295 INFO:teuthology.orchestra.run.smithi156.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:26:59.347 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:26:59.348 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2024-03-25T16:26:59.348 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:26:59.391 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install librbd1 2024-03-25T16:26:59.957 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:04:20 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:27:00.059 INFO:teuthology.orchestra.run.smithi156.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:00.110 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:27:00.111 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2024-03-25T16:27:00.112 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:27:00.157 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install python3-rados 2024-03-25T16:27:00.734 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:04:21 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:27:00.831 INFO:teuthology.orchestra.run.smithi156.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:00.882 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:27:00.883 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2024-03-25T16:27:00.883 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:27:00.928 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install python3-rgw 2024-03-25T16:27:01.499 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:04:22 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:27:01.596 INFO:teuthology.orchestra.run.smithi156.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:01.647 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:27:01.648 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2024-03-25T16:27:01.648 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:27:01.699 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install python3-cephfs 2024-03-25T16:27:02.274 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:04:23 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:27:02.370 INFO:teuthology.orchestra.run.smithi156.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:02.421 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:27:02.423 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2024-03-25T16:27:02.423 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:27:02.470 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install python3-rbd 2024-03-25T16:27:03.031 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:04:24 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:27:03.127 INFO:teuthology.orchestra.run.smithi156.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:03.179 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:27:03.180 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2024-03-25T16:27:03.180 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:27:03.223 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install rbd-fuse 2024-03-25T16:27:03.792 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:04:24 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:27:03.942 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:27:03.943 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:27:03.943 INFO:teuthology.orchestra.run.smithi156.stdout: Package Architecture Version Repository Size 2024-03-25T16:27:03.943 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:27:03.943 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T16:27:03.943 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:27:03.943 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:27:03.943 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T16:27:03.943 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:27:03.944 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-03-25T16:27:03.944 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:27:03.944 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 85 k 2024-03-25T16:27:03.944 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 231 k 2024-03-25T16:27:03.944 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T16:27:04.670 INFO:teuthology.orchestra.run.smithi156.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 117 kB/s | 85 kB 00:00 2024-03-25T16:27:04.670 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:04.670 INFO:teuthology.orchestra.run.smithi156.stdout:Total 117 kB/s | 85 kB 00:00 2024-03-25T16:27:04.670 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T16:27:04.678 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T16:27:04.678 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T16:27:04.734 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T16:27:04.735 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T16:27:04.874 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T16:27:04.984 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:05.390 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:05.699 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:05.699 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:27:05.699 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T16:27:05.699 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:05.699 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:27:05.699 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:27:05.913 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install rbd-mirror 2024-03-25T16:27:06.489 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:04:27 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:27:06.643 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:27:06.644 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:27:06.644 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-03-25T16:27:06.644 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:27:06.644 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T16:27:06.644 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:27:06.644 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:27:06.644 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T16:27:06.644 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:27:06.644 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-03-25T16:27:06.644 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:27:06.644 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 3.1 M 2024-03-25T16:27:06.644 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 13 M 2024-03-25T16:27:06.645 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T16:27:11.606 INFO:teuthology.orchestra.run.smithi156.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 637 kB/s | 3.1 MB 00:04 2024-03-25T16:27:11.607 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:11.607 INFO:teuthology.orchestra.run.smithi156.stdout:Total 637 kB/s | 3.1 MB 00:04 2024-03-25T16:27:11.608 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T16:27:11.618 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T16:27:11.619 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T16:27:11.695 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T16:27:11.696 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T16:27:12.131 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T16:27:12.199 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:12.223 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:12.223 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:27:12.223 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:27:12.223 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:27:12.223 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:27:12.223 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:27:13.340 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:13.340 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:27:13.340 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T16:27:13.341 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:13.341 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:27:13.341 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:27:13.589 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install rbd-nbd 2024-03-25T16:27:14.172 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:04:35 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:27:14.319 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:27:14.320 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:27:14.320 INFO:teuthology.orchestra.run.smithi156.stdout: Package Architecture Version Repository Size 2024-03-25T16:27:14.320 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:27:14.320 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T16:27:14.320 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:27:14.320 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:27:14.320 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T16:27:14.320 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:27:14.321 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-03-25T16:27:14.321 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:27:14.321 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 171 k 2024-03-25T16:27:14.321 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 494 k 2024-03-25T16:27:14.321 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T16:27:15.485 INFO:teuthology.orchestra.run.smithi156.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 147 kB/s | 171 kB 00:01 2024-03-25T16:27:15.486 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:15.486 INFO:teuthology.orchestra.run.smithi156.stdout:Total 147 kB/s | 171 kB 00:01 2024-03-25T16:27:15.486 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T16:27:15.496 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T16:27:15.496 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T16:27:15.557 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T16:27:15.558 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T16:27:15.712 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T16:27:15.809 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:16.250 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:16.529 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:16.530 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:27:16.530 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T16:27:16.530 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:16.530 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:27:16.530 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:27:16.753 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:27:17.354 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:04:38 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:27:17.514 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:27:17.515 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:27:17.515 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-03-25T16:27:17.515 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:27:17.515 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T16:27:17.515 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:27:17.515 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-03-25T16:27:17.515 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:27:17.515 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:27:17.515 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T16:27:17.515 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:27:17.516 INFO:teuthology.orchestra.run.smithi156.stdout:Install 2 Packages 2024-03-25T16:27:17.516 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:27:17.516 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 91 M 2024-03-25T16:27:17.516 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 498 M 2024-03-25T16:27:17.516 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T16:27:36.957 INFO:teuthology.orchestra.run.smithi194.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 513 kB/s | 22 MB 00:44 2024-03-25T16:28:01.651 INFO:teuthology.orchestra.run.smithi156.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 521 kB/s | 22 MB 00:44 2024-03-25T16:29:26.590 INFO:teuthology.orchestra.run.smithi156.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 541 kB/s | 68 MB 02:09 2024-03-25T16:29:26.590 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:26.591 INFO:teuthology.orchestra.run.smithi156.stdout:Total 719 kB/s | 91 MB 02:09 2024-03-25T16:29:26.591 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T16:29:26.595 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T16:29:26.595 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T16:29:27.174 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T16:29:27.174 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T16:29:30.305 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T16:29:37.498 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:29:37.657 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:29:41.018 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:29:41.018 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:29:41.281 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:29:41.281 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:29:41.281 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T16:29:41.281 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:41.281 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:41.282 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:29:41.282 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:29:41.504 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:29:42.072 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:07:03 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:29:42.221 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:29:42.222 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:29:42.222 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-03-25T16:29:42.222 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:29:42.222 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T16:29:42.222 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:29:42.222 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:29:42.222 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T16:29:42.222 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:29:42.222 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-03-25T16:29:42.222 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:29:42.222 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 416 M 2024-03-25T16:29:42.222 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 1.7 G 2024-03-25T16:29:42.223 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T16:30:15.375 INFO:teuthology.orchestra.run.smithi194.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 343 kB/s | 68 MB 03:23 2024-03-25T16:30:15.376 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:30:15.376 INFO:teuthology.orchestra.run.smithi194.stdout:Total 456 kB/s | 91 MB 03:23 2024-03-25T16:30:15.376 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T16:30:15.381 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T16:30:15.381 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T16:30:15.959 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T16:30:15.959 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T16:30:19.122 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T16:30:26.285 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:30:26.358 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:30:29.534 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:30:29.534 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:30:29.792 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:30:29.792 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:30:29.792 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T16:30:29.792 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:30:29.792 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:30:29.792 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:30:29.792 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:30:30.049 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:30:30.614 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:08:02 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:30:30.766 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:30:30.766 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:30:30.766 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-25T16:30:30.766 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:30:30.766 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T16:30:30.766 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:30:30.767 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:30:30.767 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T16:30:30.767 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:30:30.767 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-25T16:30:30.767 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:30:30.767 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 416 M 2024-03-25T16:30:30.767 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 1.7 G 2024-03-25T16:30:30.767 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T16:46:21.806 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 448 kB/s | 416 MB 15:51 2024-03-25T16:46:21.824 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:46:21.825 INFO:teuthology.orchestra.run.smithi194.stdout:Total 448 kB/s | 416 MB 15:51 2024-03-25T16:46:21.825 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T16:46:21.825 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T16:46:21.825 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T16:46:24.292 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T16:46:24.292 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T16:47:06.545 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T16:47:06.632 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:47:14.275 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:47:14.538 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:47:14.538 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:47:14.538 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T16:47:14.539 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:47:14.539 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:47:14.539 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:47:14.790 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:47:15.360 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:24:47 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:47:15.510 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:47:15.511 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================== 2024-03-25T16:47:15.511 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-25T16:47:15.511 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================== 2024-03-25T16:47:15.511 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T16:47:15.511 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:47:15.512 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:47:15.512 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T16:47:15.512 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================== 2024-03-25T16:47:15.512 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-25T16:47:15.512 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:47:15.512 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 1.4 M 2024-03-25T16:47:15.512 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 6.4 M 2024-03-25T16:47:15.512 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T16:47:19.902 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 333 kB/s | 1.4 MB 00:04 2024-03-25T16:47:19.903 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:47:19.903 INFO:teuthology.orchestra.run.smithi194.stdout:Total 333 kB/s | 1.4 MB 00:04 2024-03-25T16:47:19.903 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T16:47:19.907 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T16:47:19.907 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T16:47:19.951 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T16:47:19.951 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T16:47:20.228 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T16:47:20.325 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:47:20.717 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:47:20.988 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:47:20.989 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:47:20.989 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T16:47:20.989 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:47:20.989 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:47:20.989 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:47:21.175 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:47:21.737 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:24:53 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:47:21.884 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:47:21.885 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:47:21.885 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-25T16:47:21.885 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:47:21.885 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T16:47:21.885 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:47:21.885 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:47:21.885 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T16:47:21.886 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:47:21.886 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-25T16:47:21.886 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:47:21.886 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 176 M 2024-03-25T16:47:21.886 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 753 M 2024-03-25T16:47:21.886 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T16:49:33.571 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 358 kB/s | 416 MB 19:51 2024-03-25T16:49:33.572 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:49:33.572 INFO:teuthology.orchestra.run.smithi156.stdout:Total 358 kB/s | 416 MB 19:51 2024-03-25T16:49:33.572 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T16:49:33.577 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T16:49:33.578 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T16:49:36.052 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T16:49:36.052 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T16:50:18.618 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T16:50:18.708 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:50:26.867 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:50:27.131 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:50:27.131 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:50:27.131 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T16:50:27.131 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:50:27.131 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:50:27.131 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:50:27.365 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:50:27.933 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:27:48 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:50:28.083 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:50:28.083 INFO:teuthology.orchestra.run.smithi156.stdout:======================================================================================== 2024-03-25T16:50:28.083 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-03-25T16:50:28.084 INFO:teuthology.orchestra.run.smithi156.stdout:======================================================================================== 2024-03-25T16:50:28.084 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T16:50:28.084 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:50:28.084 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:50:28.084 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T16:50:28.084 INFO:teuthology.orchestra.run.smithi156.stdout:======================================================================================== 2024-03-25T16:50:28.084 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-03-25T16:50:28.084 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:50:28.084 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 1.4 M 2024-03-25T16:50:28.084 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 6.4 M 2024-03-25T16:50:28.084 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T16:50:35.293 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 203 kB/s | 1.4 MB 00:07 2024-03-25T16:50:35.294 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:50:35.294 INFO:teuthology.orchestra.run.smithi156.stdout:Total 203 kB/s | 1.4 MB 00:07 2024-03-25T16:50:35.295 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T16:50:35.299 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T16:50:35.299 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T16:50:35.345 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T16:50:35.345 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T16:50:35.612 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T16:50:35.725 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:50:36.125 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:50:36.387 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:50:36.387 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:50:36.388 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T16:50:36.388 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:50:36.388 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:50:36.388 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:50:36.573 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:50:37.166 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:27:58 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:50:37.325 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:50:37.325 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:50:37.325 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-03-25T16:50:37.325 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:50:37.326 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T16:50:37.326 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:50:37.326 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:50:37.326 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T16:50:37.326 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:50:37.326 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-03-25T16:50:37.326 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:50:37.326 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 176 M 2024-03-25T16:50:37.326 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 753 M 2024-03-25T16:50:37.326 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T16:58:11.069 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 277 kB/s | 176 MB 10:49 2024-03-25T16:58:11.069 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:58:11.069 INFO:teuthology.orchestra.run.smithi194.stdout:Total 277 kB/s | 176 MB 10:49 2024-03-25T16:58:11.069 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T16:58:11.072 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T16:58:11.072 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T16:58:12.134 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T16:58:12.134 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T16:58:30.360 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T16:58:30.572 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:58:34.948 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:58:35.208 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:58:35.208 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:58:35.208 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T16:58:35.208 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:58:35.208 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:58:35.208 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T16:58:35.411 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-test-debuginfo 2024-03-25T16:58:35.989 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:36:07 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T16:58:36.150 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T16:58:36.151 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:58:36.151 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-25T16:58:36.151 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:58:36.151 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T16:58:36.151 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T16:58:36.151 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:58:36.151 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T16:58:36.151 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T16:58:36.151 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-25T16:58:36.151 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T16:58:36.151 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 672 M 2024-03-25T16:58:36.152 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 3.1 G 2024-03-25T16:58:36.152 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T16:59:08.794 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 352 kB/s | 176 MB 08:31 2024-03-25T16:59:08.794 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:59:08.795 INFO:teuthology.orchestra.run.smithi156.stdout:Total 352 kB/s | 176 MB 08:31 2024-03-25T16:59:08.795 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T16:59:08.797 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T16:59:08.797 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T16:59:09.859 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T16:59:09.859 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T16:59:28.130 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T16:59:28.296 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:59:33.052 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:59:33.298 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:59:33.299 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:59:33.299 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T16:59:33.299 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:59:33.299 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:59:33.299 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T16:59:33.497 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-test-debuginfo 2024-03-25T16:59:34.059 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:36:55 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T16:59:34.207 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T16:59:34.208 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:59:34.208 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-03-25T16:59:34.208 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:59:34.208 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T16:59:34.209 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T16:59:34.209 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:59:34.209 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T16:59:34.209 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T16:59:34.209 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-03-25T16:59:34.209 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T16:59:34.209 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 672 M 2024-03-25T16:59:34.209 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 3.1 G 2024-03-25T16:59:34.209 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T17:34:29.724 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 320 kB/s | 672 MB 35:53 2024-03-25T17:34:29.743 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:34:29.744 INFO:teuthology.orchestra.run.smithi194.stdout:Total 320 kB/s | 672 MB 35:53 2024-03-25T17:34:29.744 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T17:34:29.744 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T17:34:29.744 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T17:34:33.711 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T17:34:33.711 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T17:35:37.232 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 318 kB/s | 672 MB 36:03 2024-03-25T17:35:37.233 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:35:37.233 INFO:teuthology.orchestra.run.smithi156.stdout:Total 318 kB/s | 672 MB 36:03 2024-03-25T17:35:37.233 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T17:35:37.238 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T17:35:37.238 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T17:35:41.208 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T17:35:41.208 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T17:35:44.670 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T17:35:44.796 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:35:54.153 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:35:54.418 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:35:54.418 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:35:54.418 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T17:35:54.418 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:35:54.418 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:35:54.418 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T17:35:54.705 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:35:55.271 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 1:13:27 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T17:35:55.367 INFO:teuthology.orchestra.run.smithi194.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:35:55.419 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T17:35:55.420 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-25T17:35:55.420 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T17:35:55.487 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:35:56.053 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 1:13:28 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T17:35:56.201 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T17:35:56.201 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T17:35:56.201 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-25T17:35:56.201 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T17:35:56.202 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T17:35:56.202 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:35:56.202 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:35:56.202 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T17:35:56.202 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T17:35:56.202 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-25T17:35:56.202 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:35:56.203 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 20 M 2024-03-25T17:35:56.203 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 103 M 2024-03-25T17:35:56.203 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T17:36:52.361 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T17:36:52.534 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:37:02.193 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:37:02.448 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:37:02.448 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:37:02.448 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T17:37:02.448 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:37:02.448 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:37:02.448 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T17:37:02.709 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:37:03.274 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 1:14:24 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T17:37:03.371 INFO:teuthology.orchestra.run.smithi156.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:37:03.423 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T17:37:03.424 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2024-03-25T17:37:03.424 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T17:37:03.483 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:37:04.087 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 1:14:25 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T17:37:04.244 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T17:37:04.245 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T17:37:04.245 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-03-25T17:37:04.245 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T17:37:04.245 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T17:37:04.245 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:37:04.245 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:37:04.245 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T17:37:04.246 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T17:37:04.246 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-03-25T17:37:04.246 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:37:04.246 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 20 M 2024-03-25T17:37:04.246 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 103 M 2024-03-25T17:37:04.246 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T17:37:04.390 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 296 kB/s | 20 MB 01:08 2024-03-25T17:37:04.391 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:37:04.392 INFO:teuthology.orchestra.run.smithi194.stdout:Total 296 kB/s | 20 MB 01:08 2024-03-25T17:37:04.392 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T17:37:04.396 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T17:37:04.396 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T17:37:04.545 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T17:37:04.546 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T17:37:06.780 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T17:37:06.880 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:37:07.825 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:37:08.082 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:37:08.082 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:37:08.082 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T17:37:08.082 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:37:08.082 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:37:08.082 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T17:37:08.266 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:37:08.833 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 1:14:40 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T17:37:08.983 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T17:37:08.983 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T17:37:08.983 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-25T17:37:08.983 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T17:37:08.983 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T17:37:08.983 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:37:08.983 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:37:08.983 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T17:37:08.983 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T17:37:08.983 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-25T17:37:08.984 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:37:08.984 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 48 M 2024-03-25T17:37:08.984 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 173 M 2024-03-25T17:37:08.984 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T17:37:46.191 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 481 kB/s | 20 MB 00:41 2024-03-25T17:37:46.191 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:37:46.192 INFO:teuthology.orchestra.run.smithi156.stdout:Total 481 kB/s | 20 MB 00:41 2024-03-25T17:37:46.192 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T17:37:46.197 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T17:37:46.197 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T17:37:46.350 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T17:37:46.351 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T17:37:48.594 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T17:37:48.764 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:37:49.718 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:37:49.974 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:37:49.975 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:37:49.975 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T17:37:49.975 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:37:49.975 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:37:49.975 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T17:37:50.158 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:37:50.729 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 1:15:11 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T17:37:50.878 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T17:37:50.879 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T17:37:50.879 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-03-25T17:37:50.879 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T17:37:50.879 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T17:37:50.879 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:37:50.879 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:37:50.880 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T17:37:50.880 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T17:37:50.880 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-03-25T17:37:50.880 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:37:50.880 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 48 M 2024-03-25T17:37:50.880 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 173 M 2024-03-25T17:37:50.880 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T17:38:45.404 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 507 kB/s | 48 MB 01:36 2024-03-25T17:38:45.405 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:38:45.405 INFO:teuthology.orchestra.run.smithi194.stdout:Total 507 kB/s | 48 MB 01:36 2024-03-25T17:38:45.405 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T17:38:45.408 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T17:38:45.408 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T17:38:45.719 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T17:38:45.719 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T17:38:50.462 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T17:38:50.618 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:38:51.917 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:38:52.178 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:38:52.179 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:38:52.179 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T17:38:52.179 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:38:52.179 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:38:52.179 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T17:38:52.366 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:38:52.932 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 1:16:24 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T17:38:53.081 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T17:38:53.082 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T17:38:53.082 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-25T17:38:53.082 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T17:38:53.082 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T17:38:53.082 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:38:53.082 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:38:53.082 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T17:38:53.082 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T17:38:53.082 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-25T17:38:53.082 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:38:53.083 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 99 M 2024-03-25T17:38:53.083 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 449 M 2024-03-25T17:38:53.083 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T17:39:59.852 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 379 kB/s | 48 MB 02:08 2024-03-25T17:39:59.853 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:39:59.853 INFO:teuthology.orchestra.run.smithi156.stdout:Total 379 kB/s | 48 MB 02:08 2024-03-25T17:39:59.853 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T17:39:59.856 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T17:39:59.856 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T17:40:00.170 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T17:40:00.170 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T17:40:04.930 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T17:40:05.050 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:40:06.403 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:40:06.668 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:40:06.668 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:40:06.669 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T17:40:06.669 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:40:06.669 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:40:06.669 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T17:40:06.841 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:40:07.415 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 1:17:28 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T17:40:07.563 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T17:40:07.565 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T17:40:07.565 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-03-25T17:40:07.565 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T17:40:07.565 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T17:40:07.565 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:40:07.565 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:40:07.566 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T17:40:07.566 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T17:40:07.566 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-03-25T17:40:07.566 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:40:07.566 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 99 M 2024-03-25T17:40:07.566 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 449 M 2024-03-25T17:40:07.566 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T17:41:58.599 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 544 kB/s | 99 MB 03:05 2024-03-25T17:41:58.600 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:41:58.600 INFO:teuthology.orchestra.run.smithi194.stdout:Total 544 kB/s | 99 MB 03:05 2024-03-25T17:41:58.600 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T17:41:58.601 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T17:41:58.601 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T17:41:59.211 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T17:41:59.212 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T17:42:09.490 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T17:42:09.592 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:42:12.374 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:42:12.621 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:42:12.621 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:42:12.621 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T17:42:12.621 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:42:12.621 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:42:12.621 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T17:42:12.806 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:42:13.369 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 1:19:45 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T17:42:13.518 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T17:42:13.519 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T17:42:13.519 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-25T17:42:13.519 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T17:42:13.519 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T17:42:13.519 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:42:13.519 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:42:13.519 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T17:42:13.519 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T17:42:13.519 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-25T17:42:13.519 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:42:13.519 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 287 M 2024-03-25T17:42:13.520 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 1.2 G 2024-03-25T17:42:13.520 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T17:44:33.684 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 379 kB/s | 99 MB 04:26 2024-03-25T17:44:33.685 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:44:33.685 INFO:teuthology.orchestra.run.smithi156.stdout:Total 379 kB/s | 99 MB 04:26 2024-03-25T17:44:33.685 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T17:44:33.688 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T17:44:33.688 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T17:44:34.296 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T17:44:34.296 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T17:44:44.563 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T17:44:44.656 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:44:47.565 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:44:47.822 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:44:47.822 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:44:47.822 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T17:44:47.822 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:44:47.822 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:44:47.822 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T17:44:48.011 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:44:48.577 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 1:22:09 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T17:44:48.726 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T17:44:48.727 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T17:44:48.727 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-03-25T17:44:48.727 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T17:44:48.727 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T17:44:48.728 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:44:48.728 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:44:48.728 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T17:44:48.728 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T17:44:48.728 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-03-25T17:44:48.728 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:44:48.728 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 287 M 2024-03-25T17:44:48.728 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 1.2 G 2024-03-25T17:44:48.728 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T17:54:35.198 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 501 kB/s | 287 MB 09:46 2024-03-25T17:54:35.198 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:54:35.198 INFO:teuthology.orchestra.run.smithi156.stdout:Total 501 kB/s | 287 MB 09:46 2024-03-25T17:54:35.199 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T17:54:35.201 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T17:54:35.201 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T17:54:36.907 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T17:54:36.907 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T17:55:06.364 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T17:55:06.919 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:55:11.477 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:55:11.790 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:55:11.790 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:55:11.790 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T17:55:11.790 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:11.790 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:55:11.790 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T17:55:11.998 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:55:12.567 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 1:32:33 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T17:55:12.714 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T17:55:12.715 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T17:55:12.715 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-03-25T17:55:12.715 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T17:55:12.715 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T17:55:12.715 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:55:12.716 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:55:12.716 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T17:55:12.716 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T17:55:12.716 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-03-25T17:55:12.716 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:55:12.716 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 5.9 M 2024-03-25T17:55:12.716 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 28 M 2024-03-25T17:55:12.716 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T17:55:25.686 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 469 kB/s | 5.9 MB 00:12 2024-03-25T17:55:25.687 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:55:25.687 INFO:teuthology.orchestra.run.smithi156.stdout:Total 469 kB/s | 5.9 MB 00:12 2024-03-25T17:55:25.688 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T17:55:25.691 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T17:55:25.692 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T17:55:25.764 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T17:55:25.765 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T17:55:26.503 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T17:55:26.610 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:55:27.179 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:55:27.444 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:55:27.444 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:55:27.444 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T17:55:27.444 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:27.445 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:55:27.445 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T17:55:27.633 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:55:28.199 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 1:32:49 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T17:55:28.349 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T17:55:28.350 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T17:55:28.350 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-03-25T17:55:28.350 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T17:55:28.350 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T17:55:28.350 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:55:28.350 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:55:28.350 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T17:55:28.351 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T17:55:28.351 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-03-25T17:55:28.351 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:55:28.351 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 109 k 2024-03-25T17:55:28.351 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 440 k 2024-03-25T17:55:28.351 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T17:55:29.115 INFO:teuthology.orchestra.run.smithi156.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 142 kB/s | 109 kB 00:00 2024-03-25T17:55:29.115 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:55:29.115 INFO:teuthology.orchestra.run.smithi156.stdout:Total 142 kB/s | 109 kB 00:00 2024-03-25T17:55:29.115 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T17:55:29.119 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T17:55:29.119 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T17:55:29.153 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T17:55:29.153 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T17:55:29.273 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T17:55:29.372 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:55:29.788 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:55:30.042 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:55:30.042 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:55:30.042 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T17:55:30.042 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:30.043 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:55:30.043 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T17:55:30.220 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:55:30.790 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 1:32:51 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T17:55:30.938 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T17:55:30.939 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T17:55:30.939 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-03-25T17:55:30.939 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T17:55:30.939 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T17:55:30.939 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:55:30.939 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:55:30.939 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T17:55:30.939 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T17:55:30.939 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-03-25T17:55:30.939 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:55:30.939 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 5.4 M 2024-03-25T17:55:30.940 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 25 M 2024-03-25T17:55:30.940 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T17:55:47.732 INFO:teuthology.orchestra.run.smithi156.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 326 kB/s | 5.4 MB 00:16 2024-03-25T17:55:47.733 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:55:47.733 INFO:teuthology.orchestra.run.smithi156.stdout:Total 326 kB/s | 5.4 MB 00:16 2024-03-25T17:55:47.733 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T17:55:47.738 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T17:55:47.738 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T17:55:47.805 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T17:55:47.805 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T17:55:48.474 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T17:55:48.581 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:55:49.128 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:55:49.389 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:55:49.389 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:55:49.389 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T17:55:49.390 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:49.390 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:55:49.390 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T17:55:49.600 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install librados2-debuginfo 2024-03-25T17:55:50.165 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 1:33:11 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T17:55:50.315 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T17:55:50.315 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T17:55:50.315 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-03-25T17:55:50.315 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T17:55:50.316 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T17:55:50.316 INFO:teuthology.orchestra.run.smithi156.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:55:50.316 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:55:50.316 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T17:55:50.316 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T17:55:50.316 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-03-25T17:55:50.316 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:55:50.316 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 44 M 2024-03-25T17:55:50.317 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 253 M 2024-03-25T17:55:50.317 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T17:57:24.614 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 322 kB/s | 287 MB 15:11 2024-03-25T17:57:24.614 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:57:24.614 INFO:teuthology.orchestra.run.smithi194.stdout:Total 322 kB/s | 287 MB 15:11 2024-03-25T17:57:24.614 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T17:57:24.617 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T17:57:24.617 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T17:57:26.321 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T17:57:26.322 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T17:57:32.138 INFO:teuthology.orchestra.run.smithi156.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 445 kB/s | 44 MB 01:41 2024-03-25T17:57:32.139 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:57:32.139 INFO:teuthology.orchestra.run.smithi156.stdout:Total 445 kB/s | 44 MB 01:41 2024-03-25T17:57:32.139 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T17:57:32.143 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T17:57:32.143 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T17:57:32.443 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T17:57:32.443 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T17:57:37.524 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T17:57:37.611 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:57:39.485 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:57:39.738 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:57:39.738 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:57:39.738 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T17:57:39.739 INFO:teuthology.orchestra.run.smithi156.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:57:39.739 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:57:39.739 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T17:57:39.961 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install librbd1-debuginfo 2024-03-25T17:57:40.526 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 1:35:01 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T17:57:40.676 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T17:57:40.676 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T17:57:40.676 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-03-25T17:57:40.677 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T17:57:40.677 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T17:57:40.677 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:57:40.677 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:57:40.677 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T17:57:40.677 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T17:57:40.677 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-03-25T17:57:40.677 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T17:57:40.677 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 52 M 2024-03-25T17:57:40.677 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 308 M 2024-03-25T17:57:40.677 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T17:57:55.852 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T17:57:56.446 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:57:59.926 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:00.212 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:00.212 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:58:00.213 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T17:58:00.213 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:00.213 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:58:00.213 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T17:58:00.408 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:58:00.975 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 1:35:32 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T17:58:01.124 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T17:58:01.125 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T17:58:01.125 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-25T17:58:01.125 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T17:58:01.125 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T17:58:01.125 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:58:01.125 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:58:01.125 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T17:58:01.125 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T17:58:01.125 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-25T17:58:01.126 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:58:01.126 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 5.9 M 2024-03-25T17:58:01.126 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 28 M 2024-03-25T17:58:01.126 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T17:58:19.093 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 339 kB/s | 5.9 MB 00:17 2024-03-25T17:58:19.093 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:19.094 INFO:teuthology.orchestra.run.smithi194.stdout:Total 339 kB/s | 5.9 MB 00:17 2024-03-25T17:58:19.094 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T17:58:19.098 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T17:58:19.098 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T17:58:19.169 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T17:58:19.170 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T17:58:19.895 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T17:58:20.007 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:58:20.592 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:58:20.849 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:58:20.849 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:58:20.849 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T17:58:20.849 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:20.849 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:58:20.849 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T17:58:21.048 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:58:21.614 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 1:35:53 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T17:58:21.762 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T17:58:21.763 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T17:58:21.763 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-25T17:58:21.763 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T17:58:21.763 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T17:58:21.764 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:58:21.764 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:58:21.764 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T17:58:21.764 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T17:58:21.764 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-25T17:58:21.764 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:58:21.764 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 109 k 2024-03-25T17:58:21.764 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 440 k 2024-03-25T17:58:21.764 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T17:58:22.453 INFO:teuthology.orchestra.run.smithi194.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 158 kB/s | 109 kB 00:00 2024-03-25T17:58:22.454 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:22.454 INFO:teuthology.orchestra.run.smithi194.stdout:Total 157 kB/s | 109 kB 00:00 2024-03-25T17:58:22.454 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T17:58:22.458 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T17:58:22.458 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T17:58:22.493 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T17:58:22.494 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T17:58:22.609 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T17:58:22.710 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:58:23.087 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:58:23.342 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:58:23.343 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:58:23.343 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T17:58:23.343 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:23.343 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:58:23.343 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T17:58:23.534 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:58:24.098 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 1:35:56 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T17:58:24.247 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T17:58:24.248 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T17:58:24.248 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-25T17:58:24.248 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T17:58:24.248 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T17:58:24.248 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:58:24.248 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:58:24.248 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T17:58:24.248 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T17:58:24.248 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-25T17:58:24.248 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:58:24.248 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 5.4 M 2024-03-25T17:58:24.248 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 25 M 2024-03-25T17:58:24.248 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T17:58:35.580 INFO:teuthology.orchestra.run.smithi194.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 484 kB/s | 5.4 MB 00:11 2024-03-25T17:58:35.581 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:35.581 INFO:teuthology.orchestra.run.smithi194.stdout:Total 484 kB/s | 5.4 MB 00:11 2024-03-25T17:58:35.581 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T17:58:35.585 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T17:58:35.585 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T17:58:35.654 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T17:58:35.654 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T17:58:36.317 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T17:58:36.422 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:58:36.927 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:58:37.189 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:58:37.190 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:58:37.190 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T17:58:37.190 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:37.190 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:58:37.190 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T17:58:37.397 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librados2-debuginfo 2024-03-25T17:58:37.963 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 1:36:09 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T17:58:38.113 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T17:58:38.114 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T17:58:38.114 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-25T17:58:38.114 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T17:58:38.114 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T17:58:38.114 INFO:teuthology.orchestra.run.smithi194.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:58:38.114 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:58:38.115 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T17:58:38.115 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T17:58:38.115 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-25T17:58:38.115 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T17:58:38.115 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 44 M 2024-03-25T17:58:38.115 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 253 M 2024-03-25T17:58:38.115 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T18:01:04.572 INFO:teuthology.orchestra.run.smithi194.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 310 kB/s | 44 MB 02:26 2024-03-25T18:01:04.572 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:04.572 INFO:teuthology.orchestra.run.smithi194.stdout:Total 310 kB/s | 44 MB 02:26 2024-03-25T18:01:04.572 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T18:01:04.575 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T18:01:04.575 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T18:01:04.875 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T18:01:04.875 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T18:01:09.914 INFO:teuthology.orchestra.run.smithi156.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 254 kB/s | 52 MB 03:29 2024-03-25T18:01:09.915 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T18:01:09.915 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:09.916 INFO:teuthology.orchestra.run.smithi156.stdout:Total 254 kB/s | 52 MB 03:29 2024-03-25T18:01:09.916 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T18:01:09.919 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T18:01:09.919 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T18:01:10.011 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:01:10.264 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T18:01:10.264 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T18:01:11.796 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:01:12.066 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:01:12.067 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:01:12.067 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T18:01:12.067 INFO:teuthology.orchestra.run.smithi194.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:12.067 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:01:12.067 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T18:01:12.261 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:01:12.844 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 1:38:44 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T18:01:12.994 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T18:01:12.995 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T18:01:12.995 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-25T18:01:12.995 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T18:01:12.995 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T18:01:12.995 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:01:12.995 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:01:12.995 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T18:01:12.995 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T18:01:12.996 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-25T18:01:12.996 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:01:12.996 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 52 M 2024-03-25T18:01:12.996 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 308 M 2024-03-25T18:01:12.996 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T18:01:16.269 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T18:01:16.363 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:01:18.546 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:01:18.800 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:01:18.800 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:01:18.800 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T18:01:18.800 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:18.800 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:01:18.801 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T18:01:19.000 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:01:19.561 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 1:38:40 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T18:01:19.710 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T18:01:19.711 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T18:01:19.711 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-03-25T18:01:19.711 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T18:01:19.711 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T18:01:19.711 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:01:19.711 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:01:19.711 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T18:01:19.711 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T18:01:19.711 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-03-25T18:01:19.712 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:01:19.712 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 615 k 2024-03-25T18:01:19.712 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 2.1 M 2024-03-25T18:01:19.712 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T18:01:21.342 INFO:teuthology.orchestra.run.smithi156.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 377 kB/s | 615 kB 00:01 2024-03-25T18:01:21.342 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:21.343 INFO:teuthology.orchestra.run.smithi156.stdout:Total 377 kB/s | 615 kB 00:01 2024-03-25T18:01:21.343 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T18:01:21.347 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T18:01:21.347 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T18:01:21.387 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T18:01:21.388 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T18:01:21.550 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T18:01:21.658 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:01:22.062 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:01:22.320 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:01:22.321 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:01:22.321 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T18:01:22.321 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:22.321 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:01:22.321 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T18:01:22.506 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:01:23.071 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 1:38:44 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T18:01:23.221 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T18:01:23.222 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T18:01:23.222 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-03-25T18:01:23.222 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T18:01:23.222 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T18:01:23.222 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:01:23.223 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:01:23.223 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T18:01:23.223 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T18:01:23.223 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-03-25T18:01:23.223 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:01:23.223 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 771 k 2024-03-25T18:01:23.223 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 2.7 M 2024-03-25T18:01:23.223 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T18:01:27.361 INFO:teuthology.orchestra.run.smithi156.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 186 kB/s | 771 kB 00:04 2024-03-25T18:01:27.361 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:27.362 INFO:teuthology.orchestra.run.smithi156.stdout:Total 186 kB/s | 771 kB 00:04 2024-03-25T18:01:27.362 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T18:01:27.366 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T18:01:27.366 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T18:01:27.407 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T18:01:27.407 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T18:01:27.592 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T18:01:27.712 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:01:28.128 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:01:28.376 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:01:28.376 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:01:28.376 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T18:01:28.376 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:28.377 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:01:28.377 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T18:01:28.591 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:01:29.159 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 1:38:50 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T18:01:29.309 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T18:01:29.310 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T18:01:29.310 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-03-25T18:01:29.310 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T18:01:29.311 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T18:01:29.311 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:01:29.311 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:01:29.311 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T18:01:29.311 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T18:01:29.311 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-03-25T18:01:29.311 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:01:29.311 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 1.1 M 2024-03-25T18:01:29.311 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 4.3 M 2024-03-25T18:01:29.311 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T18:01:35.021 INFO:teuthology.orchestra.run.smithi156.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 202 kB/s | 1.1 MB 00:05 2024-03-25T18:01:35.022 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:35.022 INFO:teuthology.orchestra.run.smithi156.stdout:Total 201 kB/s | 1.1 MB 00:05 2024-03-25T18:01:35.022 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T18:01:35.027 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T18:01:35.027 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T18:01:35.070 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T18:01:35.071 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T18:01:35.297 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T18:01:35.396 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:01:35.803 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:01:36.059 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:01:36.060 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:01:36.060 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T18:01:36.060 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:36.060 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:01:36.060 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T18:01:36.261 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:01:36.831 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 1:38:57 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T18:01:36.983 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T18:01:36.984 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T18:01:36.984 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-03-25T18:01:36.984 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T18:01:36.984 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T18:01:36.984 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:01:36.984 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:01:36.984 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T18:01:36.984 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T18:01:36.984 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-03-25T18:01:36.984 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:01:36.985 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 337 k 2024-03-25T18:01:36.985 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 1.0 M 2024-03-25T18:01:36.985 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T18:01:38.991 INFO:teuthology.orchestra.run.smithi156.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 168 kB/s | 337 kB 00:02 2024-03-25T18:01:38.992 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:38.992 INFO:teuthology.orchestra.run.smithi156.stdout:Total 168 kB/s | 337 kB 00:02 2024-03-25T18:01:38.992 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T18:01:38.996 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T18:01:38.996 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T18:01:39.036 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T18:01:39.036 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T18:01:39.172 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T18:01:39.265 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:01:39.634 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:01:39.899 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:01:39.899 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:01:39.899 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T18:01:39.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:39.900 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:01:39.900 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T18:01:40.164 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:01:40.730 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 1:39:01 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T18:01:40.879 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T18:01:40.879 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T18:01:40.880 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-03-25T18:01:40.880 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T18:01:40.880 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T18:01:40.880 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:01:40.880 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:01:40.880 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T18:01:40.880 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T18:01:40.880 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-03-25T18:01:40.880 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:01:40.881 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 509 k 2024-03-25T18:01:40.881 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 2.5 M 2024-03-25T18:01:40.881 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T18:01:42.994 INFO:teuthology.orchestra.run.smithi156.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 241 kB/s | 509 kB 00:02 2024-03-25T18:01:42.994 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:42.995 INFO:teuthology.orchestra.run.smithi156.stdout:Total 241 kB/s | 509 kB 00:02 2024-03-25T18:01:42.995 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T18:01:42.999 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T18:01:42.999 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T18:01:43.038 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T18:01:43.038 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T18:01:43.203 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T18:01:43.318 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:01:43.722 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:01:43.983 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:01:43.983 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:01:43.983 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T18:01:43.984 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:43.984 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:01:43.984 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T18:01:44.195 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:01:44.765 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 1:39:05 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T18:01:44.918 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T18:01:44.919 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T18:01:44.919 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-03-25T18:01:44.919 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T18:01:44.919 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T18:01:44.919 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:01:44.919 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:01:44.919 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T18:01:44.919 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T18:01:44.919 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-03-25T18:01:44.920 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:01:44.920 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 52 M 2024-03-25T18:01:44.920 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 298 M 2024-03-25T18:01:44.920 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T18:04:15.594 INFO:teuthology.orchestra.run.smithi156.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 350 kB/s | 52 MB 02:30 2024-03-25T18:04:15.595 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:15.595 INFO:teuthology.orchestra.run.smithi156.stdout:Total 350 kB/s | 52 MB 02:30 2024-03-25T18:04:15.595 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T18:04:15.597 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T18:04:15.597 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T18:04:15.935 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T18:04:15.935 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T18:04:21.876 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T18:04:22.049 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:04:24.123 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:04:24.382 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:04:24.382 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:04:24.382 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T18:04:24.382 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:24.382 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:04:24.382 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T18:04:24.589 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:04:24.870 INFO:teuthology.orchestra.run.smithi194.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 277 kB/s | 52 MB 03:11 2024-03-25T18:04:24.870 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:24.871 INFO:teuthology.orchestra.run.smithi194.stdout:Total 277 kB/s | 52 MB 03:11 2024-03-25T18:04:24.871 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T18:04:24.874 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T18:04:24.875 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T18:04:25.161 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 1:41:46 ago on Mon 25 Mar 2024 04:22:39 PM UTC. 2024-03-25T18:04:25.216 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T18:04:25.217 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T18:04:25.312 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-25T18:04:25.313 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T18:04:25.314 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-03-25T18:04:25.314 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T18:04:25.314 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-25T18:04:25.314 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:04:25.314 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:04:25.314 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-25T18:04:25.314 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-03-25T18:04:25.314 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-03-25T18:04:25.314 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:04:25.315 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 1.3 M 2024-03-25T18:04:25.315 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 5.5 M 2024-03-25T18:04:25.315 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-25T18:04:30.569 INFO:teuthology.orchestra.run.smithi156.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 244 kB/s | 1.3 MB 00:05 2024-03-25T18:04:30.570 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:30.570 INFO:teuthology.orchestra.run.smithi156.stdout:Total 244 kB/s | 1.3 MB 00:05 2024-03-25T18:04:30.570 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-25T18:04:30.575 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-25T18:04:30.575 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-25T18:04:30.620 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-25T18:04:30.620 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-25T18:04:30.862 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-25T18:04:30.976 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:04:31.195 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T18:04:31.389 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:04:31.418 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:04:31.664 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:04:31.664 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:04:31.664 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-25T18:04:31.664 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:31.664 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:04:31.664 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-25T18:04:31.862 DEBUG:teuthology.parallel:result is None 2024-03-25T18:04:33.525 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:04:33.775 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:04:33.775 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:04:33.775 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T18:04:33.775 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:33.775 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:04:33.775 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T18:04:33.975 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:04:34.537 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 1:42:06 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T18:04:34.687 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T18:04:34.688 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T18:04:34.688 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-25T18:04:34.688 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T18:04:34.688 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T18:04:34.688 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:04:34.688 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:04:34.688 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T18:04:34.688 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T18:04:34.688 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-25T18:04:34.688 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:04:34.688 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 615 k 2024-03-25T18:04:34.689 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 2.1 M 2024-03-25T18:04:34.689 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T18:04:39.155 INFO:teuthology.orchestra.run.smithi194.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 138 kB/s | 615 kB 00:04 2024-03-25T18:04:39.155 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:39.156 INFO:teuthology.orchestra.run.smithi194.stdout:Total 138 kB/s | 615 kB 00:04 2024-03-25T18:04:39.156 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T18:04:39.160 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T18:04:39.160 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T18:04:39.202 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T18:04:39.203 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T18:04:39.372 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T18:04:39.503 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:04:39.914 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:04:40.170 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:04:40.170 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:04:40.170 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T18:04:40.170 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:40.170 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:04:40.170 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T18:04:40.375 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:04:40.943 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 1:42:12 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T18:04:41.091 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T18:04:41.092 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T18:04:41.092 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-25T18:04:41.092 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T18:04:41.093 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T18:04:41.093 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:04:41.093 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:04:41.093 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T18:04:41.093 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T18:04:41.093 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-25T18:04:41.093 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:04:41.093 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 771 k 2024-03-25T18:04:41.093 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 2.7 M 2024-03-25T18:04:41.093 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T18:04:44.443 INFO:teuthology.orchestra.run.smithi194.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 230 kB/s | 771 kB 00:03 2024-03-25T18:04:44.443 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:44.444 INFO:teuthology.orchestra.run.smithi194.stdout:Total 230 kB/s | 771 kB 00:03 2024-03-25T18:04:44.444 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T18:04:44.448 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T18:04:44.448 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T18:04:44.490 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T18:04:44.491 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T18:04:44.690 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T18:04:44.783 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:04:45.198 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:04:45.461 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:04:45.461 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:04:45.461 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T18:04:45.461 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:45.461 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:04:45.461 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T18:04:45.693 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:04:46.263 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 1:42:18 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T18:04:46.412 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T18:04:46.413 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T18:04:46.413 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-25T18:04:46.413 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T18:04:46.413 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T18:04:46.414 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:04:46.414 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:04:46.414 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T18:04:46.414 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T18:04:46.414 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-25T18:04:46.414 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:04:46.414 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 1.1 M 2024-03-25T18:04:46.414 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 4.3 M 2024-03-25T18:04:46.414 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T18:04:53.687 INFO:teuthology.orchestra.run.smithi194.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 158 kB/s | 1.1 MB 00:07 2024-03-25T18:04:53.687 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:53.688 INFO:teuthology.orchestra.run.smithi194.stdout:Total 158 kB/s | 1.1 MB 00:07 2024-03-25T18:04:53.688 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T18:04:53.692 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T18:04:53.692 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T18:04:53.736 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T18:04:53.736 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T18:04:53.959 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T18:04:54.070 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:04:54.475 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:04:54.729 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:04:54.729 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:04:54.729 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T18:04:54.729 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:54.729 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:04:54.729 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T18:04:54.979 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:04:55.555 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 1:42:27 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T18:04:55.706 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T18:04:55.706 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T18:04:55.707 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-25T18:04:55.707 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T18:04:55.707 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T18:04:55.707 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:04:55.707 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:04:55.707 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T18:04:55.707 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T18:04:55.707 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-25T18:04:55.707 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:04:55.708 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 337 k 2024-03-25T18:04:55.708 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 1.0 M 2024-03-25T18:04:55.708 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T18:04:57.756 INFO:teuthology.orchestra.run.smithi194.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 165 kB/s | 337 kB 00:02 2024-03-25T18:04:57.756 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:57.757 INFO:teuthology.orchestra.run.smithi194.stdout:Total 165 kB/s | 337 kB 00:02 2024-03-25T18:04:57.757 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T18:04:57.761 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T18:04:57.761 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T18:04:57.800 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T18:04:57.801 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T18:04:57.943 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T18:04:58.054 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:04:58.456 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:04:58.709 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:04:58.709 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:04:58.709 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T18:04:58.709 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:58.709 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:04:58.710 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T18:04:58.936 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:04:59.498 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 1:42:31 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T18:04:59.648 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T18:04:59.649 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T18:04:59.649 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-25T18:04:59.649 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T18:04:59.649 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T18:04:59.649 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:04:59.649 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:04:59.649 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T18:04:59.649 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T18:04:59.649 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-25T18:04:59.649 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:04:59.650 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 509 k 2024-03-25T18:04:59.650 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 2.5 M 2024-03-25T18:04:59.650 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T18:05:01.776 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 239 kB/s | 509 kB 00:02 2024-03-25T18:05:01.776 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:01.777 INFO:teuthology.orchestra.run.smithi194.stdout:Total 239 kB/s | 509 kB 00:02 2024-03-25T18:05:01.777 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T18:05:01.781 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T18:05:01.781 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T18:05:01.820 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T18:05:01.820 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T18:05:01.982 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T18:05:02.100 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:05:02.496 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:05:02.759 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:05:02.759 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:05:02.759 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T18:05:02.759 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:02.759 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:05:02.759 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T18:05:02.976 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:05:03.542 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 1:42:35 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T18:05:03.693 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T18:05:03.694 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T18:05:03.694 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-25T18:05:03.694 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T18:05:03.694 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T18:05:03.694 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:05:03.694 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:05:03.694 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T18:05:03.694 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T18:05:03.695 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-25T18:05:03.695 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:05:03.695 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 52 M 2024-03-25T18:05:03.695 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 298 M 2024-03-25T18:05:03.695 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T18:07:04.913 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 435 kB/s | 52 MB 02:01 2024-03-25T18:07:04.913 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:04.913 INFO:teuthology.orchestra.run.smithi194.stdout:Total 435 kB/s | 52 MB 02:01 2024-03-25T18:07:04.913 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T18:07:04.916 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T18:07:04.917 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T18:07:05.252 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T18:07:05.252 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T18:07:11.161 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T18:07:11.267 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:07:13.239 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:07:13.493 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:07:13.493 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:13.494 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T18:07:13.494 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:13.494 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:13.494 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T18:07:13.706 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:07:14.275 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 1:44:46 ago on Mon 25 Mar 2024 04:22:28 PM UTC. 2024-03-25T18:07:14.426 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-25T18:07:14.427 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T18:07:14.427 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-25T18:07:14.427 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T18:07:14.427 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-25T18:07:14.427 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:07:14.427 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:14.427 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-25T18:07:14.427 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-25T18:07:14.428 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-25T18:07:14.428 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:14.428 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 1.3 M 2024-03-25T18:07:14.428 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 5.5 M 2024-03-25T18:07:14.428 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-25T18:07:18.763 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 296 kB/s | 1.3 MB 00:04 2024-03-25T18:07:18.764 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:18.764 INFO:teuthology.orchestra.run.smithi194.stdout:Total 296 kB/s | 1.3 MB 00:04 2024-03-25T18:07:18.764 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-25T18:07:18.768 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-25T18:07:18.768 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-25T18:07:18.812 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-25T18:07:18.812 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-25T18:07:19.055 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-25T18:07:19.155 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:19.563 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:19.828 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:19.828 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:19.828 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-25T18:07:19.828 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:19.828 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:19.828 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-25T18:07:20.035 DEBUG:teuthology.parallel:result is None 2024-03-25T18:07:20.036 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:07:20.176 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:07:20.202 INFO:teuthology.orchestra.run.smithi156.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:07:20.202 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:07:20.202 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:07:20.204 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:07:20.339 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:07:20.360 INFO:teuthology.orchestra.run.smithi194.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:07:20.360 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:07:20.360 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:07:20.361 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T18:07:20.362 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-25T18:07:20.362 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:07:20.394 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-25T18:07:20.394 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:07:20.427 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T18:07:20.427 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-25T18:07:20.427 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:07:20.465 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:07:20.534 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-25T18:07:20.535 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:07:20.561 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:07:20.630 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T18:07:20.631 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-25T18:07:20.631 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:07:20.660 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:07:20.728 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-25T18:07:20.728 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:07:20.755 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:07:20.822 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T18:07:20.822 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-25T18:07:20.822 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:07:20.850 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:07:20.918 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-25T18:07:20.919 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:07:20.945 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:07:21.012 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T18:07:21.088 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T18:07:21.089 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 777 /var/log/ceph 2024-03-25T18:07:21.091 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 777 /var/log/ceph 2024-03-25T18:07:21.117 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T18:07:21.117 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:07:21.158 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:07:21.185 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T18:07:21.185 DEBUG:teuthology.orchestra.run.smithi156:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:07:21.226 DEBUG:teuthology.orchestra.run.smithi194:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:07:21.287 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T18:07:21.287 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T18:07:21.288 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621259', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621259', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi156.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCkGg4kPQJMrq/462q94BLEniTaliNpRqetozH5/BjX8Z+YsVbRb9yqGlYNox0HbEpukRiW9mhXvMGQTIUutda4=', 'smithi194.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL8bN/RvARiv4DZo2Ft70wIhEmZyDAQhciYVLOvdDAaTOzX9xpc3r1GJHnSvl0cmcRUaESnAh3PhVN2JQjiPfsE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T18:07:21.288 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:07:21.294 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:07:21.308 DEBUG:teuthology.orchestra.run.smithi156:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:07:21.348 DEBUG:teuthology.orchestra.run.smithi194:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:07:21.375 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-25T18:07:21.375 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:07:21.428 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:07:21.429 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_1 2024-03-25T18:07:21.483 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:07:21.483 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:07:21.483 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-25T18:07:21.483 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:07:21.483 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:07:21.483 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-25 16:27:12.452250219 +0000 2024-03-25T18:07:21.483 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-25 16:21:29.160806930 +0000 2024-03-25T18:07:21.483 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-25 16:21:29.160806930 +0000 2024-03-25T18:07:21.483 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-03-25 16:21:29.160806930 +0000 2024-03-25T18:07:21.483 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:07:21.548 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-03-25T18:07:21.549 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-03-25T18:07:21.549 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000105302 s, 4.9 MB/s 2024-03-25T18:07:21.550 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:07:21.606 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_2 2024-03-25T18:07:21.661 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:07:21.661 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:07:21.661 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-25T18:07:21.661 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:07:21.661 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:07:21.661 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-25 16:27:12.452250219 +0000 2024-03-25T18:07:21.661 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-25 16:21:29.487802103 +0000 2024-03-25T18:07:21.661 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-25 16:21:29.487802103 +0000 2024-03-25T18:07:21.662 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-03-25 16:21:29.487802103 +0000 2024-03-25T18:07:21.662 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:07:21.726 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-03-25T18:07:21.727 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-03-25T18:07:21.727 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000100893 s, 5.1 MB/s 2024-03-25T18:07:21.728 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:07:21.783 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_3 2024-03-25T18:07:21.838 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:07:21.838 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:07:21.838 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-25T18:07:21.838 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:07:21.838 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:07:21.839 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-25 16:27:12.453250205 +0000 2024-03-25T18:07:21.839 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-25 16:21:29.825797113 +0000 2024-03-25T18:07:21.839 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-25 16:21:29.825797113 +0000 2024-03-25T18:07:21.839 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-03-25 16:21:29.825797113 +0000 2024-03-25T18:07:21.839 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:07:21.903 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-03-25T18:07:21.903 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-03-25T18:07:21.904 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 9.8441e-05 s, 5.2 MB/s 2024-03-25T18:07:21.905 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:07:21.961 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_4 2024-03-25T18:07:22.016 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:07:22.016 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:07:22.016 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-25T18:07:22.016 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:07:22.016 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:07:22.016 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-25 16:27:12.453250205 +0000 2024-03-25T18:07:22.016 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-25 16:21:30.150792315 +0000 2024-03-25T18:07:22.016 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-25 16:21:30.150792315 +0000 2024-03-25T18:07:22.016 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-03-25 16:21:30.150792315 +0000 2024-03-25T18:07:22.017 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:07:22.081 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-03-25T18:07:22.081 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-03-25T18:07:22.082 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000103755 s, 4.9 MB/s 2024-03-25T18:07:22.083 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:07:22.139 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:07:22.140 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-25T18:07:22.140 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:07:22.156 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:07:22.156 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_1 2024-03-25T18:07:22.212 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:07:22.213 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:07:22.213 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T18:07:22.213 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:07:22.213 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:07:22.213 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-03-25 16:26:47.531382021 +0000 2024-03-25T18:07:22.213 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-03-25 16:21:36.319113575 +0000 2024-03-25T18:07:22.213 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-03-25 16:21:36.319113575 +0000 2024-03-25T18:07:22.213 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-03-25 16:21:36.319113575 +0000 2024-03-25T18:07:22.213 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:07:22.279 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-03-25T18:07:22.280 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-03-25T18:07:22.280 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000268293 s, 1.9 MB/s 2024-03-25T18:07:22.281 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:07:22.338 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_2 2024-03-25T18:07:22.393 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:07:22.393 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:07:22.393 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-25T18:07:22.393 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:07:22.394 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:07:22.394 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-03-25 16:26:47.532381999 +0000 2024-03-25T18:07:22.394 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-03-25 16:21:36.660106069 +0000 2024-03-25T18:07:22.394 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-03-25 16:21:36.660106069 +0000 2024-03-25T18:07:22.394 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-03-25 16:21:36.660106069 +0000 2024-03-25T18:07:22.394 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:07:22.459 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-03-25T18:07:22.459 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-03-25T18:07:22.460 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000222827 s, 2.3 MB/s 2024-03-25T18:07:22.461 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:07:22.517 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_3 2024-03-25T18:07:22.572 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:07:22.572 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:07:22.573 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T18:07:22.573 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:07:22.573 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:07:22.573 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-03-25 16:26:47.532381999 +0000 2024-03-25T18:07:22.573 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-03-25 16:21:36.998098630 +0000 2024-03-25T18:07:22.573 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-03-25 16:21:36.998098630 +0000 2024-03-25T18:07:22.573 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-03-25 16:21:36.998098630 +0000 2024-03-25T18:07:22.573 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:07:22.638 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-03-25T18:07:22.638 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-03-25T18:07:22.638 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000171533 s, 3.0 MB/s 2024-03-25T18:07:22.639 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:07:22.695 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_4 2024-03-25T18:07:22.750 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:07:22.750 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:07:22.750 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-25T18:07:22.750 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:07:22.750 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:07:22.750 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-03-25 16:26:47.532381999 +0000 2024-03-25T18:07:22.750 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-03-25 16:21:37.325091433 +0000 2024-03-25T18:07:22.750 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-03-25 16:21:37.325091433 +0000 2024-03-25T18:07:22.750 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-03-25 16:21:37.325091433 +0000 2024-03-25T18:07:22.751 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:07:22.815 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-03-25T18:07:22.815 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-03-25T18:07:22.815 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000182274 s, 2.8 MB/s 2024-03-25T18:07:22.816 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:07:22.871 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:07:22.871 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi156.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi194.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:07:22.871 INFO:tasks.ceph:Generating config... 2024-03-25T18:07:22.872 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T18:07:22.872 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T18:07:22.872 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T18:07:22.872 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T18:07:22.873 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T18:07:22.873 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T18:07:22.873 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T18:07:22.873 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T18:07:22.873 INFO:tasks.ceph:[global] ms type = async 2024-03-25T18:07:22.873 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T18:07:22.873 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T18:07:22.873 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T18:07:22.873 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T18:07:22.873 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T18:07:22.873 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T18:07:22.873 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T18:07:22.873 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T18:07:22.873 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T18:07:22.874 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T18:07:22.874 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T18:07:22.874 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T18:07:22.874 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T18:07:22.874 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T18:07:22.874 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T18:07:22.874 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T18:07:22.874 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T18:07:22.874 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T18:07:22.874 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T18:07:22.874 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T18:07:22.874 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T18:07:22.874 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T18:07:22.875 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T18:07:22.875 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T18:07:22.875 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T18:07:22.875 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T18:07:22.875 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T18:07:22.875 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T18:07:22.875 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T18:07:22.875 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T18:07:22.875 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T18:07:22.875 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T18:07:22.875 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T18:07:22.875 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T18:07:22.875 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T18:07:22.875 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T18:07:22.875 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T18:07:22.876 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T18:07:22.876 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T18:07:22.876 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T18:07:22.876 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T18:07:22.876 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T18:07:22.876 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T18:07:22.876 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T18:07:22.876 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T18:07:22.876 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T18:07:22.934 INFO:teuthology.orchestra.run.smithi156.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T18:07:22.937 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T18:07:22.981 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:07:23.008 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.156:6789]'), ('mon.c', '[v1:172.21.15.156:6790]'), ('mon.b', '[v1:172.21.15.194:6789]')] 2024-03-25T18:07:23.008 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.156:6789],[v1:172.21.15.156:6790],[v1:172.21.15.194:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T18:07:23.009 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-25T18:07:23.009 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:07:23.063 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.156:6789]' --addv c '[v1:172.21.15.156:6790]' --addv b '[v1:172.21.15.194:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:23.148 INFO:teuthology.orchestra.run.smithi156.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:23.148 INFO:teuthology.orchestra.run.smithi156.stdout:monmaptool: generated fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:07:23.148 INFO:teuthology.orchestra.run.smithi156.stdout:setting min_mon_release = quincy 2024-03-25T18:07:23.148 INFO:teuthology.orchestra.run.smithi156.stdout:epoch 0 2024-03-25T18:07:23.148 INFO:teuthology.orchestra.run.smithi156.stdout:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:07:23.148 INFO:teuthology.orchestra.run.smithi156.stdout:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:07:23.148 INFO:teuthology.orchestra.run.smithi156.stdout:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:07:23.148 INFO:teuthology.orchestra.run.smithi156.stdout:min_mon_release 17 (quincy) 2024-03-25T18:07:23.148 INFO:teuthology.orchestra.run.smithi156.stdout:election_strategy: 3 2024-03-25T18:07:23.148 INFO:teuthology.orchestra.run.smithi156.stdout:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:07:23.148 INFO:teuthology.orchestra.run.smithi156.stdout:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:07:23.148 INFO:teuthology.orchestra.run.smithi156.stdout:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:07:23.149 INFO:teuthology.orchestra.run.smithi156.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T18:07:23.150 DEBUG:teuthology.orchestra.run.smithi156:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:07:23.164 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID af7d8066-e5a9-4d33-b2ab-862ad8bf139c... 2024-03-25T18:07:23.164 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:07:23.206 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:07:23.254 INFO:teuthology.orchestra.run.smithi156.stdout:[global] 2024-03-25T18:07:23.254 INFO:teuthology.orchestra.run.smithi156.stdout: chdir = "" 2024-03-25T18:07:23.254 INFO:teuthology.orchestra.run.smithi156.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:07:23.254 INFO:teuthology.orchestra.run.smithi156.stdout: auth supported = cephx 2024-03-25T18:07:23.254 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.255 INFO:teuthology.orchestra.run.smithi156.stdout: filestore xattr use omap = true 2024-03-25T18:07:23.255 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.255 INFO:teuthology.orchestra.run.smithi156.stdout: mon clock drift allowed = 1.000 2024-03-25T18:07:23.255 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.255 INFO:teuthology.orchestra.run.smithi156.stdout: osd crush chooseleaf type = 0 2024-03-25T18:07:23.255 INFO:teuthology.orchestra.run.smithi156.stdout: auth debug = true 2024-03-25T18:07:23.255 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.255 INFO:teuthology.orchestra.run.smithi156.stdout: ms die on old message = true 2024-03-25T18:07:23.255 INFO:teuthology.orchestra.run.smithi156.stdout: ms die on bug = true 2024-03-25T18:07:23.255 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.255 INFO:teuthology.orchestra.run.smithi156.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:07:23.255 INFO:teuthology.orchestra.run.smithi156.stdout: mon pg warn max object skew = 0 2024-03-25T18:07:23.255 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.255 INFO:teuthology.orchestra.run.smithi156.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:07:23.256 INFO:teuthology.orchestra.run.smithi156.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:07:23.256 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.256 INFO:teuthology.orchestra.run.smithi156.stdout: osd pool default size = 2 2024-03-25T18:07:23.256 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.256 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd allow primary affinity = true 2024-03-25T18:07:23.256 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd allow pg remap = true 2024-03-25T18:07:23.256 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:07:23.256 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:07:23.256 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on no sortbitwise = false 2024-03-25T18:07:23.256 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:07:23.256 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on too few osds = false 2024-03-25T18:07:23.256 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:07:23.256 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:07:23.256 INFO:teuthology.orchestra.run.smithi156.stdout: mon_allow_pool_size_one = true 2024-03-25T18:07:23.257 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.257 INFO:teuthology.orchestra.run.smithi156.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:07:23.257 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.257 INFO:teuthology.orchestra.run.smithi156.stdout: osd default data pool replay window = 5 2024-03-25T18:07:23.257 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.257 INFO:teuthology.orchestra.run.smithi156.stdout: mon allow pool delete = true 2024-03-25T18:07:23.257 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.257 INFO:teuthology.orchestra.run.smithi156.stdout: mon cluster log file level = debug 2024-03-25T18:07:23.257 INFO:teuthology.orchestra.run.smithi156.stdout: debug asserts on shutdown = true 2024-03-25T18:07:23.257 INFO:teuthology.orchestra.run.smithi156.stdout: mon health detail to clog = false 2024-03-25T18:07:23.257 INFO:teuthology.orchestra.run.smithi156.stdout: mon host = "[v1:172.21.15.156:6789],[v1:172.21.15.156:6790],[v1:172.21.15.194:6789]" 2024-03-25T18:07:23.257 INFO:teuthology.orchestra.run.smithi156.stdout: mon client directed command retry = 5 2024-03-25T18:07:23.257 INFO:teuthology.orchestra.run.smithi156.stdout: mon election default strategy = 3 2024-03-25T18:07:23.257 INFO:teuthology.orchestra.run.smithi156.stdout: ms bind msgr2 = False 2024-03-25T18:07:23.257 INFO:teuthology.orchestra.run.smithi156.stdout: ms inject socket failures = 5000 2024-03-25T18:07:23.258 INFO:teuthology.orchestra.run.smithi156.stdout: ms type = async 2024-03-25T18:07:23.258 INFO:teuthology.orchestra.run.smithi156.stdout: osd heartbeat grace = 80 2024-03-25T18:07:23.258 INFO:teuthology.orchestra.run.smithi156.stdout: fsid = af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:07:23.258 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.258 INFO:teuthology.orchestra.run.smithi156.stdout:[osd] 2024-03-25T18:07:23.258 INFO:teuthology.orchestra.run.smithi156.stdout: osd journal size = 100 2024-03-25T18:07:23.258 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.258 INFO:teuthology.orchestra.run.smithi156.stdout: osd scrub load threshold = 5.0 2024-03-25T18:07:23.258 INFO:teuthology.orchestra.run.smithi156.stdout: osd scrub max interval = 600 2024-03-25T18:07:23.258 INFO:teuthology.orchestra.run.smithi156.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:07:23.258 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.258 INFO:teuthology.orchestra.run.smithi156.stdout: osd recover clone overlap = true 2024-03-25T18:07:23.258 INFO:teuthology.orchestra.run.smithi156.stdout: osd recovery max chunk = 1048576 2024-03-25T18:07:23.258 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.259 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug shutdown = true 2024-03-25T18:07:23.259 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug op order = true 2024-03-25T18:07:23.259 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug verify stray on activate = true 2024-03-25T18:07:23.259 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug trim objects = true 2024-03-25T18:07:23.259 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.259 INFO:teuthology.orchestra.run.smithi156.stdout: osd open classes on start = true 2024-03-25T18:07:23.259 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug pg log writeout = true 2024-03-25T18:07:23.259 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.259 INFO:teuthology.orchestra.run.smithi156.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:07:23.259 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.259 INFO:teuthology.orchestra.run.smithi156.stdout: osd map max advance = 10 2024-03-25T18:07:23.259 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.259 INFO:teuthology.orchestra.run.smithi156.stdout: journal zero on create = true 2024-03-25T18:07:23.259 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.259 INFO:teuthology.orchestra.run.smithi156.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:07:23.259 INFO:teuthology.orchestra.run.smithi156.stdout: filestore apply finisher threads = 3 2024-03-25T18:07:23.260 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.260 INFO:teuthology.orchestra.run.smithi156.stdout: bdev debug aio = true 2024-03-25T18:07:23.260 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug misdirected ops = true 2024-03-25T18:07:23.260 INFO:teuthology.orchestra.run.smithi156.stdout: bdev async discard = True 2024-03-25T18:07:23.260 INFO:teuthology.orchestra.run.smithi156.stdout: bdev enable discard = True 2024-03-25T18:07:23.260 INFO:teuthology.orchestra.run.smithi156.stdout: bluestore allocator = stupid 2024-03-25T18:07:23.260 INFO:teuthology.orchestra.run.smithi156.stdout: bluestore block size = 96636764160 2024-03-25T18:07:23.260 INFO:teuthology.orchestra.run.smithi156.stdout: bluestore fsck on mount = True 2024-03-25T18:07:23.260 INFO:teuthology.orchestra.run.smithi156.stdout: bluestore zero block detection = True 2024-03-25T18:07:23.260 INFO:teuthology.orchestra.run.smithi156.stdout: debug bluefs = 1 2024-03-25T18:07:23.260 INFO:teuthology.orchestra.run.smithi156.stdout: debug bluestore = 1 2024-03-25T18:07:23.260 INFO:teuthology.orchestra.run.smithi156.stdout: debug monc = 20 2024-03-25T18:07:23.260 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2024-03-25T18:07:23.260 INFO:teuthology.orchestra.run.smithi156.stdout: debug osd = 20 2024-03-25T18:07:23.261 INFO:teuthology.orchestra.run.smithi156.stdout: debug rocksdb = 10 2024-03-25T18:07:23.261 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:07:23.261 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd full ratio = 0.9 2024-03-25T18:07:23.261 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:07:23.261 INFO:teuthology.orchestra.run.smithi156.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:07:23.261 INFO:teuthology.orchestra.run.smithi156.stdout: osd class default list = * 2024-03-25T18:07:23.261 INFO:teuthology.orchestra.run.smithi156.stdout: osd class load list = * 2024-03-25T18:07:23.261 INFO:teuthology.orchestra.run.smithi156.stdout: osd client watch timeout = 120 2024-03-25T18:07:23.261 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug verify cached snaps = True 2024-03-25T18:07:23.261 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug verify missing on start = True 2024-03-25T18:07:23.261 INFO:teuthology.orchestra.run.smithi156.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:07:23.261 INFO:teuthology.orchestra.run.smithi156.stdout: osd fast shutdown = False 2024-03-25T18:07:23.261 INFO:teuthology.orchestra.run.smithi156.stdout: osd max markdown count = 1000 2024-03-25T18:07:23.261 INFO:teuthology.orchestra.run.smithi156.stdout: osd mclock override recovery settings = True 2024-03-25T18:07:23.261 INFO:teuthology.orchestra.run.smithi156.stdout: osd objectstore = bluestore 2024-03-25T18:07:23.261 INFO:teuthology.orchestra.run.smithi156.stdout: osd op queue = debug_random 2024-03-25T18:07:23.262 INFO:teuthology.orchestra.run.smithi156.stdout: osd op queue cut off = debug_random 2024-03-25T18:07:23.262 INFO:teuthology.orchestra.run.smithi156.stdout: osd shutdown pgref assert = True 2024-03-25T18:07:23.262 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.262 INFO:teuthology.orchestra.run.smithi156.stdout:[mgr] 2024-03-25T18:07:23.262 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2024-03-25T18:07:23.262 INFO:teuthology.orchestra.run.smithi156.stdout: debug mgr = 20 2024-03-25T18:07:23.262 INFO:teuthology.orchestra.run.smithi156.stdout: debug mon = 20 2024-03-25T18:07:23.262 INFO:teuthology.orchestra.run.smithi156.stdout: debug auth = 20 2024-03-25T18:07:23.262 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:07:23.262 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:07:23.262 INFO:teuthology.orchestra.run.smithi156.stdout: mgr/telemetry/nag = false 2024-03-25T18:07:23.262 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.262 INFO:teuthology.orchestra.run.smithi156.stdout:[mon] 2024-03-25T18:07:23.262 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2024-03-25T18:07:23.263 INFO:teuthology.orchestra.run.smithi156.stdout: debug mon = 20 2024-03-25T18:07:23.263 INFO:teuthology.orchestra.run.smithi156.stdout: debug paxos = 20 2024-03-25T18:07:23.263 INFO:teuthology.orchestra.run.smithi156.stdout: debug auth = 20 2024-03-25T18:07:23.263 INFO:teuthology.orchestra.run.smithi156.stdout: mon data avail warn = 5 2024-03-25T18:07:23.263 INFO:teuthology.orchestra.run.smithi156.stdout: mon mgr mkfs grace = 240 2024-03-25T18:07:23.263 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:07:23.263 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd reporter subtree level = osd 2024-03-25T18:07:23.263 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd prime pg temp = true 2024-03-25T18:07:23.263 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:07:23.263 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.263 INFO:teuthology.orchestra.run.smithi156.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:07:23.263 INFO:teuthology.orchestra.run.smithi156.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:07:23.263 INFO:teuthology.orchestra.run.smithi156.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:07:23.263 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.263 INFO:teuthology.orchestra.run.smithi156.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:07:23.263 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:07:23.264 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:07:23.264 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.264 INFO:teuthology.orchestra.run.smithi156.stdout: # 1m isn't quite enough 2024-03-25T18:07:23.264 INFO:teuthology.orchestra.run.smithi156.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:07:23.264 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.264 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:07:23.264 INFO:teuthology.orchestra.run.smithi156.stdout: mon min osdmap epochs = 50 2024-03-25T18:07:23.264 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd crush smoke test = False 2024-03-25T18:07:23.264 INFO:teuthology.orchestra.run.smithi156.stdout: mon osdmap full prune interval = 2 2024-03-25T18:07:23.264 INFO:teuthology.orchestra.run.smithi156.stdout: mon osdmap full prune min = 15 2024-03-25T18:07:23.264 INFO:teuthology.orchestra.run.smithi156.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:07:23.264 INFO:teuthology.orchestra.run.smithi156.stdout: mon scrub interval = 300 2024-03-25T18:07:23.264 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on pool no app = False 2024-03-25T18:07:23.264 INFO:teuthology.orchestra.run.smithi156.stdout: paxos service trim min = 10 2024-03-25T18:07:23.264 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:07:23.265 INFO:teuthology.orchestra.run.smithi156.stdout:[client] 2024-03-25T18:07:23.265 INFO:teuthology.orchestra.run.smithi156.stdout: rgw cache enabled = true 2024-03-25T18:07:23.265 INFO:teuthology.orchestra.run.smithi156.stdout: rgw enable ops log = true 2024-03-25T18:07:23.265 INFO:teuthology.orchestra.run.smithi156.stdout: rgw enable usage log = true 2024-03-25T18:07:23.265 INFO:teuthology.orchestra.run.smithi156.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:07:23.265 INFO:teuthology.orchestra.run.smithi156.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:07:23.265 INFO:teuthology.orchestra.run.smithi156.stdout: debug monc = 20 2024-03-25T18:07:23.265 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2024-03-25T18:07:23.265 INFO:teuthology.orchestra.run.smithi156.stdout: debug objecter = 20 2024-03-25T18:07:23.265 INFO:teuthology.orchestra.run.smithi156.stdout: debug rados = 20 2024-03-25T18:07:23.265 INFO:teuthology.orchestra.run.smithi156.stdout:[mon.a] 2024-03-25T18:07:23.265 INFO:teuthology.orchestra.run.smithi156.stdout:[mon.c] 2024-03-25T18:07:23.265 INFO:teuthology.orchestra.run.smithi156.stdout:[mon.b] 2024-03-25T18:07:23.266 INFO:teuthology.orchestra.run.smithi194.stdout:[global] 2024-03-25T18:07:23.266 INFO:teuthology.orchestra.run.smithi194.stdout: chdir = "" 2024-03-25T18:07:23.266 INFO:teuthology.orchestra.run.smithi194.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:07:23.266 INFO:teuthology.orchestra.run.smithi194.stdout: auth supported = cephx 2024-03-25T18:07:23.266 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.266 INFO:teuthology.orchestra.run.smithi194.stdout: filestore xattr use omap = true 2024-03-25T18:07:23.266 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.266 INFO:teuthology.orchestra.run.smithi194.stdout: mon clock drift allowed = 1.000 2024-03-25T18:07:23.266 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.266 INFO:teuthology.orchestra.run.smithi194.stdout: osd crush chooseleaf type = 0 2024-03-25T18:07:23.266 INFO:teuthology.orchestra.run.smithi194.stdout: auth debug = true 2024-03-25T18:07:23.266 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.267 INFO:teuthology.orchestra.run.smithi194.stdout: ms die on old message = true 2024-03-25T18:07:23.267 INFO:teuthology.orchestra.run.smithi194.stdout: ms die on bug = true 2024-03-25T18:07:23.267 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.267 INFO:teuthology.orchestra.run.smithi194.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:07:23.267 INFO:teuthology.orchestra.run.smithi194.stdout: mon pg warn max object skew = 0 2024-03-25T18:07:23.267 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.267 INFO:teuthology.orchestra.run.smithi194.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:07:23.267 INFO:teuthology.orchestra.run.smithi194.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:07:23.267 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.267 INFO:teuthology.orchestra.run.smithi194.stdout: osd pool default size = 2 2024-03-25T18:07:23.267 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.267 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd allow primary affinity = true 2024-03-25T18:07:23.267 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd allow pg remap = true 2024-03-25T18:07:23.267 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:07:23.268 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:07:23.268 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on no sortbitwise = false 2024-03-25T18:07:23.268 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:07:23.268 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on too few osds = false 2024-03-25T18:07:23.268 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:07:23.268 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:07:23.268 INFO:teuthology.orchestra.run.smithi194.stdout: mon_allow_pool_size_one = true 2024-03-25T18:07:23.268 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.268 INFO:teuthology.orchestra.run.smithi194.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:07:23.268 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.268 INFO:teuthology.orchestra.run.smithi194.stdout: osd default data pool replay window = 5 2024-03-25T18:07:23.268 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.268 INFO:teuthology.orchestra.run.smithi194.stdout: mon allow pool delete = true 2024-03-25T18:07:23.268 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.268 INFO:teuthology.orchestra.run.smithi194.stdout: mon cluster log file level = debug 2024-03-25T18:07:23.269 INFO:teuthology.orchestra.run.smithi194.stdout: debug asserts on shutdown = true 2024-03-25T18:07:23.269 INFO:teuthology.orchestra.run.smithi194.stdout: mon health detail to clog = false 2024-03-25T18:07:23.269 INFO:teuthology.orchestra.run.smithi194.stdout: mon host = "[v1:172.21.15.156:6789],[v1:172.21.15.156:6790],[v1:172.21.15.194:6789]" 2024-03-25T18:07:23.269 INFO:teuthology.orchestra.run.smithi194.stdout: mon client directed command retry = 5 2024-03-25T18:07:23.269 INFO:teuthology.orchestra.run.smithi194.stdout: mon election default strategy = 3 2024-03-25T18:07:23.269 INFO:teuthology.orchestra.run.smithi194.stdout: ms bind msgr2 = False 2024-03-25T18:07:23.269 INFO:teuthology.orchestra.run.smithi194.stdout: ms inject socket failures = 5000 2024-03-25T18:07:23.269 INFO:teuthology.orchestra.run.smithi194.stdout: ms type = async 2024-03-25T18:07:23.269 INFO:teuthology.orchestra.run.smithi194.stdout: osd heartbeat grace = 80 2024-03-25T18:07:23.269 INFO:teuthology.orchestra.run.smithi194.stdout: fsid = af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:07:23.269 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.269 INFO:teuthology.orchestra.run.smithi194.stdout:[osd] 2024-03-25T18:07:23.269 INFO:teuthology.orchestra.run.smithi194.stdout: osd journal size = 100 2024-03-25T18:07:23.269 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.270 INFO:teuthology.orchestra.run.smithi194.stdout: osd scrub load threshold = 5.0 2024-03-25T18:07:23.270 INFO:teuthology.orchestra.run.smithi194.stdout: osd scrub max interval = 600 2024-03-25T18:07:23.270 INFO:teuthology.orchestra.run.smithi194.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:07:23.270 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.270 INFO:teuthology.orchestra.run.smithi194.stdout: osd recover clone overlap = true 2024-03-25T18:07:23.270 INFO:teuthology.orchestra.run.smithi194.stdout: osd recovery max chunk = 1048576 2024-03-25T18:07:23.270 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.270 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug shutdown = true 2024-03-25T18:07:23.270 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug op order = true 2024-03-25T18:07:23.270 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug verify stray on activate = true 2024-03-25T18:07:23.270 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug trim objects = true 2024-03-25T18:07:23.270 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.270 INFO:teuthology.orchestra.run.smithi194.stdout: osd open classes on start = true 2024-03-25T18:07:23.270 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug pg log writeout = true 2024-03-25T18:07:23.271 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.271 INFO:teuthology.orchestra.run.smithi194.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:07:23.271 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.271 INFO:teuthology.orchestra.run.smithi194.stdout: osd map max advance = 10 2024-03-25T18:07:23.271 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.271 INFO:teuthology.orchestra.run.smithi194.stdout: journal zero on create = true 2024-03-25T18:07:23.271 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.271 INFO:teuthology.orchestra.run.smithi194.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:07:23.271 INFO:teuthology.orchestra.run.smithi194.stdout: filestore apply finisher threads = 3 2024-03-25T18:07:23.271 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.271 INFO:teuthology.orchestra.run.smithi194.stdout: bdev debug aio = true 2024-03-25T18:07:23.271 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug misdirected ops = true 2024-03-25T18:07:23.271 INFO:teuthology.orchestra.run.smithi194.stdout: bdev async discard = True 2024-03-25T18:07:23.271 INFO:teuthology.orchestra.run.smithi194.stdout: bdev enable discard = True 2024-03-25T18:07:23.271 INFO:teuthology.orchestra.run.smithi194.stdout: bluestore allocator = stupid 2024-03-25T18:07:23.271 INFO:teuthology.orchestra.run.smithi194.stdout: bluestore block size = 96636764160 2024-03-25T18:07:23.272 INFO:teuthology.orchestra.run.smithi194.stdout: bluestore fsck on mount = True 2024-03-25T18:07:23.272 INFO:teuthology.orchestra.run.smithi194.stdout: bluestore zero block detection = True 2024-03-25T18:07:23.272 INFO:teuthology.orchestra.run.smithi194.stdout: debug bluefs = 1 2024-03-25T18:07:23.272 INFO:teuthology.orchestra.run.smithi194.stdout: debug bluestore = 1 2024-03-25T18:07:23.272 INFO:teuthology.orchestra.run.smithi194.stdout: debug monc = 20 2024-03-25T18:07:23.272 INFO:teuthology.orchestra.run.smithi194.stdout: debug ms = 1 2024-03-25T18:07:23.272 INFO:teuthology.orchestra.run.smithi194.stdout: debug osd = 20 2024-03-25T18:07:23.272 INFO:teuthology.orchestra.run.smithi194.stdout: debug rocksdb = 10 2024-03-25T18:07:23.272 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:07:23.272 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd full ratio = 0.9 2024-03-25T18:07:23.272 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:07:23.272 INFO:teuthology.orchestra.run.smithi194.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:07:23.272 INFO:teuthology.orchestra.run.smithi194.stdout: osd class default list = * 2024-03-25T18:07:23.272 INFO:teuthology.orchestra.run.smithi194.stdout: osd class load list = * 2024-03-25T18:07:23.272 INFO:teuthology.orchestra.run.smithi194.stdout: osd client watch timeout = 120 2024-03-25T18:07:23.272 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug verify cached snaps = True 2024-03-25T18:07:23.272 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug verify missing on start = True 2024-03-25T18:07:23.273 INFO:teuthology.orchestra.run.smithi194.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:07:23.273 INFO:teuthology.orchestra.run.smithi194.stdout: osd fast shutdown = False 2024-03-25T18:07:23.273 INFO:teuthology.orchestra.run.smithi194.stdout: osd max markdown count = 1000 2024-03-25T18:07:23.273 INFO:teuthology.orchestra.run.smithi194.stdout: osd mclock override recovery settings = True 2024-03-25T18:07:23.273 INFO:teuthology.orchestra.run.smithi194.stdout: osd objectstore = bluestore 2024-03-25T18:07:23.273 INFO:teuthology.orchestra.run.smithi194.stdout: osd op queue = debug_random 2024-03-25T18:07:23.273 INFO:teuthology.orchestra.run.smithi194.stdout: osd op queue cut off = debug_random 2024-03-25T18:07:23.273 INFO:teuthology.orchestra.run.smithi194.stdout: osd shutdown pgref assert = True 2024-03-25T18:07:23.273 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.273 INFO:teuthology.orchestra.run.smithi194.stdout:[mgr] 2024-03-25T18:07:23.273 INFO:teuthology.orchestra.run.smithi194.stdout: debug ms = 1 2024-03-25T18:07:23.273 INFO:teuthology.orchestra.run.smithi194.stdout: debug mgr = 20 2024-03-25T18:07:23.273 INFO:teuthology.orchestra.run.smithi194.stdout: debug mon = 20 2024-03-25T18:07:23.273 INFO:teuthology.orchestra.run.smithi194.stdout: debug auth = 20 2024-03-25T18:07:23.273 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:07:23.274 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:07:23.274 INFO:teuthology.orchestra.run.smithi194.stdout: mgr/telemetry/nag = false 2024-03-25T18:07:23.274 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.274 INFO:teuthology.orchestra.run.smithi194.stdout:[mon] 2024-03-25T18:07:23.274 INFO:teuthology.orchestra.run.smithi194.stdout: debug ms = 1 2024-03-25T18:07:23.274 INFO:teuthology.orchestra.run.smithi194.stdout: debug mon = 20 2024-03-25T18:07:23.274 INFO:teuthology.orchestra.run.smithi194.stdout: debug paxos = 20 2024-03-25T18:07:23.274 INFO:teuthology.orchestra.run.smithi194.stdout: debug auth = 20 2024-03-25T18:07:23.274 INFO:teuthology.orchestra.run.smithi194.stdout: mon data avail warn = 5 2024-03-25T18:07:23.274 INFO:teuthology.orchestra.run.smithi194.stdout: mon mgr mkfs grace = 240 2024-03-25T18:07:23.274 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:07:23.274 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd reporter subtree level = osd 2024-03-25T18:07:23.274 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd prime pg temp = true 2024-03-25T18:07:23.274 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:07:23.275 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.275 INFO:teuthology.orchestra.run.smithi194.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:07:23.275 INFO:teuthology.orchestra.run.smithi194.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:07:23.275 INFO:teuthology.orchestra.run.smithi194.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:07:23.275 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.275 INFO:teuthology.orchestra.run.smithi194.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:07:23.275 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:07:23.275 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:07:23.275 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.275 INFO:teuthology.orchestra.run.smithi194.stdout: # 1m isn't quite enough 2024-03-25T18:07:23.275 INFO:teuthology.orchestra.run.smithi194.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:07:23.275 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.275 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:07:23.275 INFO:teuthology.orchestra.run.smithi194.stdout: mon min osdmap epochs = 50 2024-03-25T18:07:23.276 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd crush smoke test = False 2024-03-25T18:07:23.276 INFO:teuthology.orchestra.run.smithi194.stdout: mon osdmap full prune interval = 2 2024-03-25T18:07:23.276 INFO:teuthology.orchestra.run.smithi194.stdout: mon osdmap full prune min = 15 2024-03-25T18:07:23.276 INFO:teuthology.orchestra.run.smithi194.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:07:23.276 INFO:teuthology.orchestra.run.smithi194.stdout: mon scrub interval = 300 2024-03-25T18:07:23.276 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on pool no app = False 2024-03-25T18:07:23.276 INFO:teuthology.orchestra.run.smithi194.stdout: paxos service trim min = 10 2024-03-25T18:07:23.276 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-25T18:07:23.276 INFO:teuthology.orchestra.run.smithi194.stdout:[client] 2024-03-25T18:07:23.276 INFO:teuthology.orchestra.run.smithi194.stdout: rgw cache enabled = true 2024-03-25T18:07:23.276 INFO:teuthology.orchestra.run.smithi194.stdout: rgw enable ops log = true 2024-03-25T18:07:23.276 INFO:teuthology.orchestra.run.smithi194.stdout: rgw enable usage log = true 2024-03-25T18:07:23.276 INFO:teuthology.orchestra.run.smithi194.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:07:23.276 INFO:teuthology.orchestra.run.smithi194.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:07:23.277 INFO:teuthology.orchestra.run.smithi194.stdout: debug monc = 20 2024-03-25T18:07:23.277 INFO:teuthology.orchestra.run.smithi194.stdout: debug ms = 1 2024-03-25T18:07:23.277 INFO:teuthology.orchestra.run.smithi194.stdout: debug objecter = 20 2024-03-25T18:07:23.277 INFO:teuthology.orchestra.run.smithi194.stdout: debug rados = 20 2024-03-25T18:07:23.277 INFO:teuthology.orchestra.run.smithi194.stdout:[mon.a] 2024-03-25T18:07:23.277 INFO:teuthology.orchestra.run.smithi194.stdout:[mon.c] 2024-03-25T18:07:23.277 INFO:teuthology.orchestra.run.smithi194.stdout:[mon.b] 2024-03-25T18:07:23.278 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T18:07:23.278 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T18:07:23.332 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T18:07:23.332 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-25T18:07:23.332 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T18:07:23.347 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-25T18:07:23.347 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T18:07:23.401 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi156.front.sepia.ceph.com 2024-03-25T18:07:23.402 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-25T18:07:23.402 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:07:23.402 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:07:23.480 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-25T18:07:23.480 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:23.535 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi194.front.sepia.ceph.com 2024-03-25T18:07:23.535 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-25T18:07:23.535 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:07:23.535 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:07:23.574 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-25T18:07:23.574 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:23.629 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T18:07:23.629 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T18:07:23.629 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:07:23.683 INFO:teuthology.orchestra.run.smithi156.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:07:23.686 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:07:23.749 INFO:teuthology.orchestra.run.smithi194.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:07:23.751 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T18:07:23.751 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T18:07:23.752 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T18:07:23.793 INFO:teuthology.orchestra.run.smithi156.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T18:07:23.809 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T18:07:23.851 INFO:teuthology.orchestra.run.smithi194.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T18:07:23.866 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T18:07:23.866 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi156.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi194.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:07:23.866 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T18:07:23.893 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:07:23.893 INFO:tasks.ceph:role: osd.0 2024-03-25T18:07:23.893 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi156.front.sepia.ceph.com 2024-03-25T18:07:23.893 DEBUG:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:07:23.959 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:07:23.960 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:07:23.960 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:07:23.960 INFO:teuthology.orchestra.run.smithi156.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:07:23.960 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:07:23.960 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:07:23.960 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:07:23.960 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:07:23.960 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:07:23.960 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:07:23.992 INFO:teuthology.orchestra.run.smithi156.stdout:Discarding blocks...Done. 2024-03-25T18:07:23.994 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2024-03-25T18:07:23.994 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T18:07:24.066 DEBUG:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T18:07:24.138 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T18:07:24.204 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:07:24.204 INFO:tasks.ceph:role: osd.1 2024-03-25T18:07:24.204 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi156.front.sepia.ceph.com 2024-03-25T18:07:24.204 DEBUG:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:07:24.270 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:07:24.271 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:07:24.271 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:07:24.271 INFO:teuthology.orchestra.run.smithi156.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:07:24.271 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:07:24.271 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:07:24.271 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:07:24.271 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:07:24.271 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:07:24.271 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:07:24.302 INFO:teuthology.orchestra.run.smithi156.stdout:Discarding blocks...Done. 2024-03-25T18:07:24.304 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2024-03-25T18:07:24.304 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T18:07:24.375 DEBUG:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T18:07:24.446 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T18:07:24.512 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:07:24.512 INFO:tasks.ceph:role: osd.2 2024-03-25T18:07:24.512 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi156.front.sepia.ceph.com 2024-03-25T18:07:24.512 DEBUG:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:07:24.577 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:07:24.577 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:07:24.578 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:07:24.578 INFO:teuthology.orchestra.run.smithi156.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:07:24.578 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:07:24.578 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:07:24.578 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:07:24.578 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:07:24.578 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:07:24.578 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:07:24.610 INFO:teuthology.orchestra.run.smithi156.stdout:Discarding blocks...Done. 2024-03-25T18:07:24.612 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2024-03-25T18:07:24.612 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T18:07:24.682 DEBUG:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T18:07:24.753 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T18:07:24.820 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:07:24.820 INFO:tasks.ceph:role: osd.3 2024-03-25T18:07:24.820 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi156.front.sepia.ceph.com 2024-03-25T18:07:24.820 DEBUG:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:07:24.887 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:07:24.887 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:07:24.887 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:07:24.887 INFO:teuthology.orchestra.run.smithi156.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:07:24.887 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:07:24.887 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:07:24.887 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:07:24.887 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:07:24.887 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:07:24.887 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:07:24.918 INFO:teuthology.orchestra.run.smithi156.stdout:Discarding blocks...Done. 2024-03-25T18:07:24.920 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2024-03-25T18:07:24.921 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T18:07:24.992 DEBUG:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T18:07:25.062 DEBUG:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:25.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:25.155+0000 7f496f813640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T18:07:25.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:25.155+0000 7f496f813640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T18:07:25.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:25.155+0000 7f496f813640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:07:25.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:25.155+0000 7f496f813640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:07:25.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:25.155+0000 7f496f813640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:07:25.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:25.155+0000 7f496f813640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T18:07:27.997 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T18:07:28.029 DEBUG:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:28.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:28.120+0000 7f12a8260640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T18:07:28.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:28.120+0000 7f12a8260640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T18:07:28.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:28.120+0000 7f12a8260640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:07:28.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:28.120+0000 7f12a8260640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:07:28.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:28.120+0000 7f12a8260640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:07:28.121 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:28.120+0000 7f12a8260640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T18:07:30.954 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T18:07:30.986 DEBUG:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:31.079 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:31.078+0000 7feeaaa0f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T18:07:31.079 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:31.078+0000 7feeaaa0f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T18:07:31.079 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:31.078+0000 7feeaaa0f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:07:31.079 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:31.078+0000 7feeaaa0f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:07:31.079 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:31.078+0000 7feeaaa0f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:07:31.079 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:31.078+0000 7feeaaa0f640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T18:07:33.914 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T18:07:33.946 DEBUG:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:34.036 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:34.035+0000 7f75ea89d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T18:07:34.036 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:34.035+0000 7f75ea89d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T18:07:34.036 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:34.035+0000 7f75ea89d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:07:34.036 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:34.035+0000 7f75ea89d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:07:34.036 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:34.035+0000 7f75ea89d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:07:34.036 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:34.035+0000 7f75ea89d640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T18:07:36.877 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T18:07:36.907 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T18:07:36.939 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:07:36.939 INFO:tasks.ceph:role: osd.4 2024-03-25T18:07:36.939 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi194.front.sepia.ceph.com 2024-03-25T18:07:36.939 DEBUG:teuthology.orchestra.run.smithi194:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:07:37.016 INFO:teuthology.orchestra.run.smithi194.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:07:37.016 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:07:37.016 INFO:teuthology.orchestra.run.smithi194.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:07:37.016 INFO:teuthology.orchestra.run.smithi194.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:07:37.016 INFO:teuthology.orchestra.run.smithi194.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:07:37.016 INFO:teuthology.orchestra.run.smithi194.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:07:37.016 INFO:teuthology.orchestra.run.smithi194.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:07:37.016 INFO:teuthology.orchestra.run.smithi194.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:07:37.016 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:07:37.016 INFO:teuthology.orchestra.run.smithi194.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:07:37.530 INFO:teuthology.orchestra.run.smithi194.stdout:Discarding blocks...Done. 2024-03-25T18:07:37.532 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi194.front.sepia.ceph.com -o noatime 2024-03-25T18:07:37.532 DEBUG:teuthology.orchestra.run.smithi194:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T18:07:37.683 DEBUG:teuthology.orchestra.run.smithi194:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T18:07:37.719 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T18:07:37.789 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:07:37.789 INFO:tasks.ceph:role: osd.5 2024-03-25T18:07:37.789 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi194.front.sepia.ceph.com 2024-03-25T18:07:37.789 DEBUG:teuthology.orchestra.run.smithi194:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:07:37.865 INFO:teuthology.orchestra.run.smithi194.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:07:37.865 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:07:37.865 INFO:teuthology.orchestra.run.smithi194.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:07:37.865 INFO:teuthology.orchestra.run.smithi194.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:07:37.865 INFO:teuthology.orchestra.run.smithi194.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:07:37.865 INFO:teuthology.orchestra.run.smithi194.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:07:37.865 INFO:teuthology.orchestra.run.smithi194.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:07:37.865 INFO:teuthology.orchestra.run.smithi194.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:07:37.866 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:07:37.866 INFO:teuthology.orchestra.run.smithi194.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:07:38.321 INFO:teuthology.orchestra.run.smithi194.stdout:Discarding blocks...Done. 2024-03-25T18:07:38.323 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi194.front.sepia.ceph.com -o noatime 2024-03-25T18:07:38.323 DEBUG:teuthology.orchestra.run.smithi194:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T18:07:38.417 DEBUG:teuthology.orchestra.run.smithi194:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T18:07:38.450 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T18:07:38.519 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:07:38.519 INFO:tasks.ceph:role: osd.6 2024-03-25T18:07:38.519 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi194.front.sepia.ceph.com 2024-03-25T18:07:38.519 DEBUG:teuthology.orchestra.run.smithi194:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:07:38.595 INFO:teuthology.orchestra.run.smithi194.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:07:38.595 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:07:38.595 INFO:teuthology.orchestra.run.smithi194.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:07:38.596 INFO:teuthology.orchestra.run.smithi194.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:07:38.596 INFO:teuthology.orchestra.run.smithi194.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:07:38.596 INFO:teuthology.orchestra.run.smithi194.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:07:38.596 INFO:teuthology.orchestra.run.smithi194.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:07:38.596 INFO:teuthology.orchestra.run.smithi194.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:07:38.596 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:07:38.596 INFO:teuthology.orchestra.run.smithi194.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:07:39.080 INFO:teuthology.orchestra.run.smithi194.stdout:Discarding blocks...Done. 2024-03-25T18:07:39.082 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi194.front.sepia.ceph.com -o noatime 2024-03-25T18:07:39.082 DEBUG:teuthology.orchestra.run.smithi194:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T18:07:39.233 DEBUG:teuthology.orchestra.run.smithi194:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T18:07:39.268 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T18:07:39.337 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:07:39.337 INFO:tasks.ceph:role: osd.7 2024-03-25T18:07:39.337 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi194.front.sepia.ceph.com 2024-03-25T18:07:39.337 DEBUG:teuthology.orchestra.run.smithi194:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:07:39.413 INFO:teuthology.orchestra.run.smithi194.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:07:39.413 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:07:39.413 INFO:teuthology.orchestra.run.smithi194.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:07:39.413 INFO:teuthology.orchestra.run.smithi194.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:07:39.413 INFO:teuthology.orchestra.run.smithi194.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:07:39.413 INFO:teuthology.orchestra.run.smithi194.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:07:39.413 INFO:teuthology.orchestra.run.smithi194.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:07:39.413 INFO:teuthology.orchestra.run.smithi194.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:07:39.413 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:07:39.414 INFO:teuthology.orchestra.run.smithi194.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:07:39.890 INFO:teuthology.orchestra.run.smithi194.stdout:Discarding blocks...Done. 2024-03-25T18:07:39.891 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi194.front.sepia.ceph.com -o noatime 2024-03-25T18:07:39.892 DEBUG:teuthology.orchestra.run.smithi194:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T18:07:40.031 DEBUG:teuthology.orchestra.run.smithi194:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T18:07:40.065 DEBUG:teuthology.orchestra.run.smithi194:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:40.158 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:07:40.158+0000 7fed7220f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T18:07:40.159 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:07:40.158+0000 7fed7220f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T18:07:40.159 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:07:40.158+0000 7fed7220f640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:07:40.159 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:07:40.158+0000 7fed7220f640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:07:40.159 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:07:40.158+0000 7fed7220f640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:07:40.159 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:07:40.159+0000 7fed7220f640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T18:07:43.000 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T18:07:43.029 DEBUG:teuthology.orchestra.run.smithi194:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:43.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:07:43.118+0000 7efdf440d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T18:07:43.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:07:43.118+0000 7efdf440d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T18:07:43.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:07:43.118+0000 7efdf440d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:07:43.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:07:43.118+0000 7efdf440d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:07:43.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:07:43.118+0000 7efdf440d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:07:43.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:07:43.119+0000 7efdf440d640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T18:07:45.957 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T18:07:45.987 DEBUG:teuthology.orchestra.run.smithi194:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:46.077 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:07:46.076+0000 7fc451a11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T18:07:46.077 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:07:46.077+0000 7fc451a11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T18:07:46.078 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:07:46.077+0000 7fc451a11640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:07:46.078 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:07:46.077+0000 7fc451a11640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:07:46.078 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:07:46.077+0000 7fc451a11640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:07:46.078 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:07:46.077+0000 7fc451a11640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T18:07:48.918 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T18:07:48.947 DEBUG:teuthology.orchestra.run.smithi194:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:49.037 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:07:49.036+0000 7f49ebc68640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T18:07:49.037 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:07:49.036+0000 7f49ebc68640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T18:07:49.037 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:07:49.036+0000 7f49ebc68640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:07:49.038 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:07:49.036+0000 7f49ebc68640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:07:49.038 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:07:49.036+0000 7f49ebc68640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:07:49.038 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:07:49.037+0000 7f49ebc68640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T18:07:51.874 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T18:07:51.905 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T18:07:51.906 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-25T18:07:51.906 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T18:07:51.938 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-25T18:07:51.938 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T18:07:52.008 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-25T18:07:52.008 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T18:07:52.078 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-25T18:07:52.078 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T18:07:52.147 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-25T18:07:52.147 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T18:07:52.216 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-25T18:07:52.216 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T18:07:52.247 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-25T18:07:52.247 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T18:07:52.317 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-25T18:07:52.317 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T18:07:52.386 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-25T18:07:52.386 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T18:07:52.454 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-25T18:07:52.454 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T18:07:52.522 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-25T18:07:52.523 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T18:07:52.539 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-25T18:07:52.539 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T18:07:52.577 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T18:07:52.578 DEBUG:teuthology.orchestra.run.smithi156:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:07:52.582 DEBUG:teuthology.orchestra.run.smithi194:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:07:52.620 INFO:teuthology.orchestra.run.smithi156.stdout:[mgr.y] 2024-03-25T18:07:52.620 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQBbvQFmV/DLKBAA+AJpKQrQHrfQ+WD4Rl8caQ== 2024-03-25T18:07:52.620 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.0] 2024-03-25T18:07:52.621 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQBdvQFmadBRCRAALBxBDoYwqOexY8VjLMvjqA== 2024-03-25T18:07:52.621 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.1] 2024-03-25T18:07:52.621 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQBgvQFm/2s7BxAADy73/yNHOHfrlT4gioDSaA== 2024-03-25T18:07:52.621 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.2] 2024-03-25T18:07:52.621 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQBjvQFmg9m4BBAAsLU+IDLp5eEOuSAOzTvzPw== 2024-03-25T18:07:52.621 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.3] 2024-03-25T18:07:52.621 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQBmvQFmi3YoAhAAZojf1IdCNMtBd1ff94Rvmw== 2024-03-25T18:07:52.621 INFO:teuthology.orchestra.run.smithi156.stdout:[mgr.x] 2024-03-25T18:07:52.621 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQBbvQFmpgGyLBAAdlntnMx5AHhi+sdSNvPZ1g== 2024-03-25T18:07:52.621 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.4] 2024-03-25T18:07:52.621 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQBsvQFmlod5CRAArow4UTEnEPF2nedh7Jefzw== 2024-03-25T18:07:52.621 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.5] 2024-03-25T18:07:52.621 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQBvvQFmMvUcBxAA4U6xrO+9yV/95ENffbPf/g== 2024-03-25T18:07:52.621 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.6] 2024-03-25T18:07:52.621 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQByvQFmV/iiBBAA1MamOLAURbn8d/oWL9c1UA== 2024-03-25T18:07:52.622 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.7] 2024-03-25T18:07:52.622 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQB1vQFmQMA6AhAAp6l+qnQ3wt0augPYndM9uw== 2024-03-25T18:07:52.622 INFO:teuthology.orchestra.run.smithi156.stdout:[client.0] 2024-03-25T18:07:52.622 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQBbvQFmH6BZLxAAbRpo6sUe4spb/6yQfmh8xg== 2024-03-25T18:07:52.622 INFO:teuthology.orchestra.run.smithi156.stdout:[client.1] 2024-03-25T18:07:52.622 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQBbvQFmPHTDMhAAeX+6BFMl6O4O1ywAr+QsIQ== 2024-03-25T18:07:52.643 INFO:teuthology.orchestra.run.smithi194.stdout:[mgr.y] 2024-03-25T18:07:52.643 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBbvQFmV/DLKBAA+AJpKQrQHrfQ+WD4Rl8caQ== 2024-03-25T18:07:52.643 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.0] 2024-03-25T18:07:52.643 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBdvQFmadBRCRAALBxBDoYwqOexY8VjLMvjqA== 2024-03-25T18:07:52.643 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.1] 2024-03-25T18:07:52.643 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBgvQFm/2s7BxAADy73/yNHOHfrlT4gioDSaA== 2024-03-25T18:07:52.643 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.2] 2024-03-25T18:07:52.643 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBjvQFmg9m4BBAAsLU+IDLp5eEOuSAOzTvzPw== 2024-03-25T18:07:52.644 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.3] 2024-03-25T18:07:52.644 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBmvQFmi3YoAhAAZojf1IdCNMtBd1ff94Rvmw== 2024-03-25T18:07:52.644 INFO:teuthology.orchestra.run.smithi194.stdout:[mgr.x] 2024-03-25T18:07:52.644 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBbvQFmpgGyLBAAdlntnMx5AHhi+sdSNvPZ1g== 2024-03-25T18:07:52.644 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.4] 2024-03-25T18:07:52.644 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBsvQFmlod5CRAArow4UTEnEPF2nedh7Jefzw== 2024-03-25T18:07:52.644 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.5] 2024-03-25T18:07:52.644 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBvvQFmMvUcBxAA4U6xrO+9yV/95ENffbPf/g== 2024-03-25T18:07:52.644 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.6] 2024-03-25T18:07:52.644 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQByvQFmV/iiBBAA1MamOLAURbn8d/oWL9c1UA== 2024-03-25T18:07:52.644 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.7] 2024-03-25T18:07:52.644 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQB1vQFmQMA6AhAAp6l+qnQ3wt0augPYndM9uw== 2024-03-25T18:07:52.644 INFO:teuthology.orchestra.run.smithi194.stdout:[client.0] 2024-03-25T18:07:52.644 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBbvQFmH6BZLxAAbRpo6sUe4spb/6yQfmh8xg== 2024-03-25T18:07:52.644 INFO:teuthology.orchestra.run.smithi194.stdout:[client.1] 2024-03-25T18:07:52.645 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBbvQFmPHTDMhAAeX+6BFMl6O4O1ywAr+QsIQ== 2024-03-25T18:07:52.645 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:07:52.666 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:07:52.737 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:52.739 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:52.791 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:52.793 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:52.844 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:52.847 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:52.898 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:52.901 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:52.952 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:07:52.954 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:07:53.006 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:53.008 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:53.059 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:53.062 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:53.113 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:53.115 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:53.166 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:53.168 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:53.219 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:07:53.222 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:07:53.276 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:07:53.278 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:07:53.332 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T18:07:53.332 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T18:07:53.359 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:07:53.456 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T18:07:53.482 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T18:07:53.549 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:07:53.647 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T18:07:53.673 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T18:07:53.701 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:07:53.800 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T18:07:53.827 DEBUG:teuthology.orchestra.run.smithi156:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:53.829 DEBUG:teuthology.orchestra.run.smithi194:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:53.882 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T18:07:53.882 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:07:53.882 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T18:07:53.882 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T18:07:53.884 INFO:tasks.ceph.mon.a:Started 2024-03-25T18:07:53.885 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:07:53.885 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T18:07:53.885 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T18:07:53.887 INFO:tasks.ceph.mon.c:Started 2024-03-25T18:07:53.887 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:07:53.887 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T18:07:53.887 DEBUG:teuthology.orchestra.run.smithi194:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T18:07:53.925 INFO:tasks.ceph.mon.b:Started 2024-03-25T18:07:53.925 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T18:07:53.925 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T18:07:53.925 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T18:07:53.927 INFO:tasks.ceph.mgr.y:Started 2024-03-25T18:07:53.927 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T18:07:53.928 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T18:07:53.929 INFO:tasks.ceph.mgr.x:Started 2024-03-25T18:07:53.929 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T18:07:53.930 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T18:07:54.031 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:54.030+0000 7fca7ae43640 10 monclient: get_monmap_and_config 2024-03-25T18:07:54.031 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:54.030+0000 7fca7ae43640 10 monclient: build_initial_monmap 2024-03-25T18:07:54.031 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:54.030+0000 7fca7ae43640 10 monclient: monmap: 2024-03-25T18:07:54.031 INFO:teuthology.orchestra.run.smithi156.stderr:epoch 0 2024-03-25T18:07:54.031 INFO:teuthology.orchestra.run.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:07:54.031 INFO:teuthology.orchestra.run.smithi156.stderr:last_changed 2024-03-25T18:07:54.031820+0000 2024-03-25T18:07:54.032 INFO:teuthology.orchestra.run.smithi156.stderr:created 2024-03-25T18:07:54.031820+0000 2024-03-25T18:07:54.032 INFO:teuthology.orchestra.run.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:54.032 INFO:teuthology.orchestra.run.smithi156.stderr:election_strategy: 3 2024-03-25T18:07:54.032 INFO:teuthology.orchestra.run.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:07:54.032 INFO:teuthology.orchestra.run.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:07:54.032 INFO:teuthology.orchestra.run.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:07:54.032 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-25T18:07:54.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:54.031+0000 7fca7ae43640 1 Processor -- start 2024-03-25T18:07:54.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:54.031+0000 7fca7ae43640 1 -- start start 2024-03-25T18:07:54.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:54.031+0000 7fca7ae43640 10 monclient: init 2024-03-25T18:07:54.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:54.032+0000 7fca7ae43640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:54.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:54.032+0000 7fca7ae43640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:07:54.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:54.032+0000 7fca7ae43640 10 monclient(hunting): picked mon.noname-a con 0x7fca740a9920 addr v1:172.21.15.156:6789/0 2024-03-25T18:07:54.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:54.032+0000 7fca7ae43640 10 monclient(hunting): picked mon.noname-c con 0x7fca740b45a0 addr v1:172.21.15.194:6789/0 2024-03-25T18:07:54.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:54.032+0000 7fca7ae43640 10 monclient(hunting): picked mon.noname-b con 0x7fca74146450 addr v1:172.21.15.156:6790/0 2024-03-25T18:07:54.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:54.032+0000 7fca7ae43640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7406b100 con 0x7fca740a9920 2024-03-25T18:07:54.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:54.032+0000 7fca7ae43640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca740b2940 con 0x7fca740b45a0 2024-03-25T18:07:54.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:54.032+0000 7fca7ae43640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7413fec0 con 0x7fca74146450 2024-03-25T18:07:54.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:54.032+0000 7fca7ae43640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:54.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:54.032+0000 7fca7ae43640 10 monclient(hunting): authenticate will time out at 6955.343018s 2024-03-25T18:07:54.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:54.032+0000 7fca78bdf640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7fca740a9920 legacy=0x7fca7413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2024-03-25T18:07:54.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:54.032+0000 7fca793e0640 1 -- >> v1:172.21.15.156:6790/0 conn(0x7fca74146450 legacy=0x7fca7414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6790/0 2024-03-25T18:07:54.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:54.032+0000 7fca73fff640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7fca740b45a0 legacy=0x7fca74145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-03-25T18:07:54.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:54.233+0000 7fca78bdf640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7fca740a9920 legacy=0x7fca7413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2024-03-25T18:07:54.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:54.233+0000 7fca793e0640 1 -- >> v1:172.21.15.156:6790/0 conn(0x7fca74146450 legacy=0x7fca7414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6790/0 2024-03-25T18:07:54.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:54.233+0000 7fca73fff640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7fca740b45a0 legacy=0x7fca74145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-03-25T18:07:54.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:54.634+0000 7fca78bdf640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7fca740a9920 legacy=0x7fca7413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2024-03-25T18:07:54.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:54.634+0000 7fca793e0640 1 -- >> v1:172.21.15.156:6790/0 conn(0x7fca74146450 legacy=0x7fca7414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6790/0 2024-03-25T18:07:54.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:54.634+0000 7fca73fff640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7fca740b45a0 legacy=0x7fca74145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-03-25T18:07:55.436 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:55.435+0000 7fca78bdf640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7fca740a9920 legacy=0x7fca7413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2024-03-25T18:07:55.436 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:55.435+0000 7fca793e0640 1 -- >> v1:172.21.15.156:6790/0 conn(0x7fca74146450 legacy=0x7fca7414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6790/0 2024-03-25T18:07:55.436 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:55.435+0000 7fca73fff640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7fca740b45a0 legacy=0x7fca74145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-03-25T18:07:57.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:57.032+0000 7fca727fc640 10 monclient(hunting): tick 2024-03-25T18:07:57.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:57.032+0000 7fca727fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:07:57.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:57.032+0000 7fca727fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:07:57.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:57.032+0000 7fca727fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:07:57.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:57.032+0000 7fca727fc640 1 -- >> v1:172.21.15.156:6790/0 conn(0x7fca74146450 legacy=0x7fca7414dc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:07:57.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:57.032+0000 7fca727fc640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7fca740b45a0 legacy=0x7fca74145d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:07:57.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:57.032+0000 7fca727fc640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7fca740a9920 legacy=0x7fca7413f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:07:57.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:57.032+0000 7fca727fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:07:57.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:57.032+0000 7fca727fc640 10 monclient(hunting): picked mon.noname-b con 0x7fca74146450 addr v1:172.21.15.156:6790/0 2024-03-25T18:07:57.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:57.032+0000 7fca727fc640 10 monclient(hunting): picked mon.noname-c con 0x7fca740b45a0 addr v1:172.21.15.194:6789/0 2024-03-25T18:07:57.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:57.032+0000 7fca727fc640 10 monclient(hunting): picked mon.noname-a con 0x7fca740a9920 addr v1:172.21.15.156:6789/0 2024-03-25T18:07:57.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:57.032+0000 7fca727fc640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7406b100 con 0x7fca740a9920 2024-03-25T18:07:57.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:57.032+0000 7fca78bdf640 1 -- >> v1:172.21.15.156:6790/0 conn(0x7fca74146450 legacy=0x7fca580056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6790/0 2024-03-25T18:07:57.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:57.032+0000 7fca727fc640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca740b2940 con 0x7fca740b45a0 2024-03-25T18:07:57.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:57.032+0000 7fca727fc640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7413fec0 con 0x7fca74146450 2024-03-25T18:07:57.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:57.032+0000 7fca793e0640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7fca740a9920 legacy=0x7fca5800c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2024-03-25T18:07:57.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:57.032+0000 7fca727fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:57.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:57.032+0000 7fca73fff640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7fca740b45a0 legacy=0x7fca58008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-03-25T18:07:57.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:57.233+0000 7fca793e0640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7fca740a9920 legacy=0x7fca5800c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2024-03-25T18:07:57.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:57.233+0000 7fca78bdf640 1 -- >> v1:172.21.15.156:6790/0 conn(0x7fca74146450 legacy=0x7fca580056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6790/0 2024-03-25T18:07:57.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:57.233+0000 7fca73fff640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7fca740b45a0 legacy=0x7fca58008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-03-25T18:07:57.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:57.634+0000 7fca793e0640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7fca740a9920 legacy=0x7fca5800c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2024-03-25T18:07:57.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:57.634+0000 7fca78bdf640 1 -- >> v1:172.21.15.156:6790/0 conn(0x7fca74146450 legacy=0x7fca580056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6790/0 2024-03-25T18:07:57.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:57.634+0000 7fca73fff640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7fca740b45a0 legacy=0x7fca58008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-03-25T18:07:58.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:58.435+0000 7fca793e0640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7fca740a9920 legacy=0x7fca5800c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2024-03-25T18:07:58.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:58.435+0000 7fca78bdf640 1 -- >> v1:172.21.15.156:6790/0 conn(0x7fca74146450 legacy=0x7fca580056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6790/0 2024-03-25T18:07:58.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:07:58.435+0000 7fca73fff640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7fca740b45a0 legacy=0x7fca58008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-03-25T18:08:00.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:00.032+0000 7fca727fc640 10 monclient(hunting): tick 2024-03-25T18:08:00.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:00.032+0000 7fca727fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:08:00.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:00.032+0000 7fca727fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:08:00.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:00.032+0000 7fca727fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:08:00.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:00.032+0000 7fca727fc640 1 -- >> v1:172.21.15.156:6790/0 conn(0x7fca74146450 legacy=0x7fca580056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:08:00.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:00.032+0000 7fca727fc640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7fca740b45a0 legacy=0x7fca58008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:08:00.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:00.032+0000 7fca727fc640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7fca740a9920 legacy=0x7fca5800c670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:08:00.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:00.032+0000 7fca727fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:08:00.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:00.032+0000 7fca727fc640 10 monclient(hunting): picked mon.noname-a con 0x7fca740a9920 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:00.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:00.032+0000 7fca727fc640 10 monclient(hunting): picked mon.noname-b con 0x7fca74146450 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:00.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:00.032+0000 7fca727fc640 10 monclient(hunting): picked mon.noname-c con 0x7fca740b45a0 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:00.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:00.032+0000 7fca727fc640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca58010e20 con 0x7fca740a9920 2024-03-25T18:08:00.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:00.032+0000 7fca727fc640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7406b100 con 0x7fca740b45a0 2024-03-25T18:08:00.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:00.032+0000 7fca727fc640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca740b2940 con 0x7fca74146450 2024-03-25T18:08:00.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:00.032+0000 7fca727fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:00.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:00.032+0000 7fca73fff640 1 -- >> v1:172.21.15.156:6790/0 conn(0x7fca74146450 legacy=0x7fca580056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6790/0 2024-03-25T18:08:00.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:00.032+0000 7fca78bdf640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7fca740a9920 legacy=0x7fca5800fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2024-03-25T18:08:00.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:00.032+0000 7fca793e0640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7fca740b45a0 legacy=0x7fca58008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-03-25T18:08:00.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:00.233+0000 7fca73fff640 1 -- >> v1:172.21.15.156:6790/0 conn(0x7fca74146450 legacy=0x7fca580056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6790/0 2024-03-25T18:08:00.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:00.233+0000 7fca78bdf640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7fca740a9920 legacy=0x7fca5800fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2024-03-25T18:08:00.234 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:00.233+0000 7fca793e0640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7fca740b45a0 legacy=0x7fca58008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-03-25T18:08:00.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:00.634+0000 7fca73fff640 1 -- >> v1:172.21.15.156:6790/0 conn(0x7fca74146450 legacy=0x7fca580056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6790/0 2024-03-25T18:08:00.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:00.634+0000 7fca78bdf640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7fca740a9920 legacy=0x7fca5800fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2024-03-25T18:08:00.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:00.634+0000 7fca793e0640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7fca740b45a0 legacy=0x7fca58008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-03-25T18:08:00.878 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:00:06.845 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:01.139 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:00:07.130 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:01.140 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:00:07.131 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:01.184 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:00:07.152 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:01.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:01.435+0000 7fca78bdf640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7fca740a9920 legacy=0x7fca5800fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2024-03-25T18:08:01.436 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:01.435+0000 7fca73fff640 1 -- >> v1:172.21.15.156:6790/0 conn(0x7fca74146450 legacy=0x7fca580056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6790/0 2024-03-25T18:08:01.436 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:01.435+0000 7fca793e0640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7fca740b45a0 legacy=0x7fca58008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-03-25T18:08:01.459 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:00:07.450 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:01.460 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:00:07.451 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:03.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:03.032+0000 7fca727fc640 10 monclient(hunting): tick 2024-03-25T18:08:03.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:03.032+0000 7fca727fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:08:03.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:03.032+0000 7fca727fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:08:03.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:03.032+0000 7fca727fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:08:03.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:03.032+0000 7fca727fc640 1 -- >> v1:172.21.15.156:6790/0 conn(0x7fca74146450 legacy=0x7fca580056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:08:03.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:03.032+0000 7fca727fc640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7fca740b45a0 legacy=0x7fca58008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:08:03.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:03.032+0000 7fca727fc640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7fca740a9920 legacy=0x7fca5800fdf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:08:03.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:03.032+0000 7fca727fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:08:03.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:03.032+0000 7fca727fc640 10 monclient(hunting): picked mon.noname-a con 0x7fca740a9920 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:03.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:03.032+0000 7fca727fc640 10 monclient(hunting): picked mon.noname-b con 0x7fca74146450 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:03.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:03.032+0000 7fca727fc640 10 monclient(hunting): picked mon.noname-c con 0x7fca740b45a0 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:03.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:03.032+0000 7fca727fc640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7413fec0 con 0x7fca740a9920 2024-03-25T18:08:03.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:03.032+0000 7fca727fc640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca58010e20 con 0x7fca740b45a0 2024-03-25T18:08:03.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:03.032+0000 7fca727fc640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7406b100 con 0x7fca74146450 2024-03-25T18:08:03.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:03.032+0000 7fca727fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:03.086 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:03.086+0000 7fca793e0640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fca740b45a0 0x7fca58008e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:52680/0 (socket says 172.21.15.156:52680) 2024-03-25T18:08:03.086 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:03.086+0000 7fca793e0640 1 -- 172.21.15.156:0/342947227 learned_addr learned my addr 172.21.15.156:0/342947227 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:08:06.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:06.033+0000 7fca727fc640 10 monclient(hunting): tick 2024-03-25T18:08:06.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:06.033+0000 7fca727fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:08:06.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:06.033+0000 7fca727fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:08:06.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:06.033+0000 7fca727fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:08:06.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:06.033+0000 7fca727fc640 1 -- 172.21.15.156:0/342947227 >> v1:172.21.15.156:6790/0 conn(0x7fca74146450 legacy=0x7fca5800f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:06.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:06.033+0000 7fca727fc640 1 -- 172.21.15.156:0/342947227 >> v1:172.21.15.194:6789/0 conn(0x7fca740b45a0 legacy=0x7fca58008e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:06.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:06.033+0000 7fca727fc640 1 -- 172.21.15.156:0/342947227 >> v1:172.21.15.156:6789/0 conn(0x7fca740a9920 legacy=0x7fca580105e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:06.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:06.033+0000 7fca727fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:08:06.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:06.033+0000 7fca727fc640 10 monclient(hunting): picked mon.noname-b con 0x7fca5800fdd0 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:06.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:06.033+0000 7fca727fc640 10 monclient(hunting): picked mon.noname-c con 0x7fca740b45a0 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:06.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:06.033+0000 7fca727fc640 10 monclient(hunting): picked mon.noname-a con 0x7fca740a9920 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:06.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:06.033+0000 7fca727fc640 1 -- 172.21.15.156:0/342947227 --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca740b2940 con 0x7fca740a9920 2024-03-25T18:08:06.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:06.033+0000 7fca727fc640 1 -- 172.21.15.156:0/342947227 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca58010330 con 0x7fca740b45a0 2024-03-25T18:08:06.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:06.033+0000 7fca727fc640 1 -- 172.21.15.156:0/342947227 --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca58010680 con 0x7fca5800fdd0 2024-03-25T18:08:06.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:06.033+0000 7fca727fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:09.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:09.033+0000 7fca727fc640 10 monclient(hunting): tick 2024-03-25T18:08:09.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:09.033+0000 7fca727fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:08:09.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:09.033+0000 7fca727fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:08:09.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:09.033+0000 7fca727fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:08:09.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:09.033+0000 7fca727fc640 1 -- 172.21.15.156:0/342947227 >> v1:172.21.15.156:6790/0 conn(0x7fca5800fdd0 legacy=0x7fca580056b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:09.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:09.033+0000 7fca727fc640 1 -- 172.21.15.156:0/342947227 >> v1:172.21.15.194:6789/0 conn(0x7fca740b45a0 legacy=0x7fca5800f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:09.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:09.033+0000 7fca727fc640 1 -- 172.21.15.156:0/342947227 >> v1:172.21.15.156:6789/0 conn(0x7fca740a9920 legacy=0x7fca58008e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:09.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:09.033+0000 7fca727fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:08:09.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:09.033+0000 7fca727fc640 10 monclient(hunting): picked mon.noname-c con 0x7fca5800a5e0 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:09.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:09.033+0000 7fca727fc640 10 monclient(hunting): picked mon.noname-b con 0x7fca5800fdd0 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:09.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:09.033+0000 7fca727fc640 10 monclient(hunting): picked mon.noname-a con 0x7fca740a9920 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:09.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:09.033+0000 7fca727fc640 1 -- 172.21.15.156:0/342947227 --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca580108b0 con 0x7fca740a9920 2024-03-25T18:08:09.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:09.033+0000 7fca727fc640 1 -- 172.21.15.156:0/342947227 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca5800f6c0 con 0x7fca5800a5e0 2024-03-25T18:08:09.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:09.033+0000 7fca727fc640 1 -- 172.21.15.156:0/342947227 --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca5800f8f0 con 0x7fca5800fdd0 2024-03-25T18:08:09.036 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:09.033+0000 7fca727fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:12.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:12.034+0000 7fca727fc640 10 monclient(hunting): tick 2024-03-25T18:08:12.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:12.034+0000 7fca727fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:08:12.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:12.034+0000 7fca727fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:08:12.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:12.034+0000 7fca727fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:08:12.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:12.034+0000 7fca727fc640 1 -- 172.21.15.156:0/342947227 >> v1:172.21.15.156:6790/0 conn(0x7fca5800fdd0 legacy=0x7fca580056b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:12.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:12.034+0000 7fca727fc640 1 -- 172.21.15.156:0/342947227 >> v1:172.21.15.194:6789/0 conn(0x7fca5800a5e0 legacy=0x7fca5800ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:12.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:12.034+0000 7fca727fc640 1 -- 172.21.15.156:0/342947227 >> v1:172.21.15.156:6789/0 conn(0x7fca740a9920 legacy=0x7fca58008e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:12.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:12.034+0000 7fca727fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:08:12.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:12.034+0000 7fca727fc640 10 monclient(hunting): picked mon.noname-c con 0x7fca580095d0 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:12.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:12.034+0000 7fca727fc640 10 monclient(hunting): picked mon.noname-b con 0x7fca5800fdd0 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:12.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:12.034+0000 7fca727fc640 10 monclient(hunting): picked mon.noname-a con 0x7fca740a9920 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:12.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:12.034+0000 7fca727fc640 1 -- 172.21.15.156:0/342947227 --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca5800fb20 con 0x7fca740a9920 2024-03-25T18:08:12.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:12.034+0000 7fca727fc640 1 -- 172.21.15.156:0/342947227 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca5800a1c0 con 0x7fca580095d0 2024-03-25T18:08:12.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:12.034+0000 7fca727fc640 1 -- 172.21.15.156:0/342947227 --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca5800a3f0 con 0x7fca5800fdd0 2024-03-25T18:08:12.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:12.034+0000 7fca727fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:15.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:15.034+0000 7fca727fc640 10 monclient(hunting): tick 2024-03-25T18:08:15.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:15.034+0000 7fca727fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:08:15.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:15.034+0000 7fca727fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:08:15.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:15.034+0000 7fca727fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:08:15.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:15.034+0000 7fca727fc640 1 -- 172.21.15.156:0/342947227 >> v1:172.21.15.156:6790/0 conn(0x7fca5800fdd0 legacy=0x7fca580056b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:15.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:15.034+0000 7fca727fc640 1 -- 172.21.15.156:0/342947227 >> v1:172.21.15.194:6789/0 conn(0x7fca580095d0 legacy=0x7fca58009a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:15.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:15.034+0000 7fca727fc640 1 -- 172.21.15.156:0/342947227 >> v1:172.21.15.156:6789/0 conn(0x7fca740a9920 legacy=0x7fca58008e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:15.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:15.034+0000 7fca727fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:08:15.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:15.034+0000 7fca727fc640 10 monclient(hunting): picked mon.noname-c con 0x7fca580120a0 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:15.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:15.034+0000 7fca727fc640 10 monclient(hunting): picked mon.noname-b con 0x7fca5800fdd0 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:15.035 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:15.034+0000 7fca727fc640 10 monclient(hunting): picked mon.noname-a con 0x7fca740a9920 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:15.036 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:15.034+0000 7fca727fc640 1 -- 172.21.15.156:0/342947227 --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca5800f1a0 con 0x7fca740a9920 2024-03-25T18:08:15.036 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:15.034+0000 7fca727fc640 1 -- 172.21.15.156:0/342947227 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca5800f3f0 con 0x7fca580120a0 2024-03-25T18:08:15.036 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:15.034+0000 7fca727fc640 1 -- 172.21.15.156:0/342947227 --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca580134f0 con 0x7fca5800fdd0 2024-03-25T18:08:15.036 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:15.034+0000 7fca727fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:17.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.086+0000 7fca737fe640 1 -- 172.21.15.156:0/342947227 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3472716118 0 0) 0x7fca5800f1a0 con 0x7fca740a9920 2024-03-25T18:08:17.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.086+0000 7fca737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:17.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.086+0000 7fca737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:17.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.086+0000 7fca737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:17.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.086+0000 7fca737fe640 10 monclient(hunting): my global_id is 4101 2024-03-25T18:08:17.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.086+0000 7fca737fe640 1 -- 172.21.15.156:0/342947227 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca64003520 con 0x7fca740a9920 2024-03-25T18:08:17.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.118+0000 7fca737fe640 1 -- 172.21.15.156:0/342947227 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3140392070 0 0) 0x7fca580134f0 con 0x7fca5800fdd0 2024-03-25T18:08:17.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.118+0000 7fca737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:17.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.118+0000 7fca737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:17.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.118+0000 7fca737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:17.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.118+0000 7fca737fe640 10 monclient(hunting): my global_id is 4106 2024-03-25T18:08:17.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.118+0000 7fca737fe640 1 -- 172.21.15.156:0/342947227 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca5800f1a0 con 0x7fca5800fdd0 2024-03-25T18:08:17.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.133+0000 7fca737fe640 1 -- 172.21.15.156:0/342947227 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3304312101 0 0) 0x7fca64003520 con 0x7fca740a9920 2024-03-25T18:08:17.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.134+0000 7fca737fe640 1 -- 172.21.15.156:0/342947227 >> v1:172.21.15.156:6790/0 conn(0x7fca5800fdd0 legacy=0x7fca580099b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:17.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.134+0000 7fca737fe640 1 -- 172.21.15.156:0/342947227 >> v1:172.21.15.194:6789/0 conn(0x7fca580120a0 legacy=0x7fca5800ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:17.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.134+0000 7fca737fe640 10 monclient: _finish_hunting 0 2024-03-25T18:08:17.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.134+0000 7fca737fe640 1 monclient: found mon.noname-a 2024-03-25T18:08:17.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.134+0000 7fca737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:17.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.134+0000 7fca737fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:08:17.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.134+0000 7fca737fe640 1 -- 172.21.15.156:0/342947227 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca58008f10 con 0x7fca740a9920 2024-03-25T18:08:17.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.134+0000 7fca737fe640 10 monclient: _finish_auth 0 2024-03-25T18:08:17.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.134+0000 7fca737fe640 10 monclient: _check_auth_tickets 2024-03-25T18:08:17.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.134+0000 7fca737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:17.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.134+0000 7fca737fe640 1 -- 172.21.15.156:0/342947227 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fca6c004220 con 0x7fca740a9920 2024-03-25T18:08:17.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.134+0000 7fca737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:17.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.134+0000 7fca737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:08:17.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.134+0000 7fca737fe640 10 monclient: dump: 2024-03-25T18:08:17.136 INFO:teuthology.orchestra.run.smithi156.stderr:epoch 1 2024-03-25T18:08:17.136 INFO:teuthology.orchestra.run.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:17.136 INFO:teuthology.orchestra.run.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:17.136 INFO:teuthology.orchestra.run.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:17.136 INFO:teuthology.orchestra.run.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:17.136 INFO:teuthology.orchestra.run.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:17.136 INFO:teuthology.orchestra.run.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:17.136 INFO:teuthology.orchestra.run.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:17.137 INFO:teuthology.orchestra.run.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:17.137 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-25T18:08:17.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.134+0000 7fca7ae43640 5 monclient: authenticate success, global_id 4101 2024-03-25T18:08:17.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.134+0000 7fca7ae43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:17.174 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.173+0000 7fca737fe640 1 -- 172.21.15.156:0/342947227 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca6c0043f0 con 0x7fca740a9920 2024-03-25T18:08:17.174 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.173+0000 7fca737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:17.174 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.173+0000 7fca737fe640 1 -- 172.21.15.156:0/342947227 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fca6c004800 con 0x7fca740a9920 2024-03-25T18:08:17.174 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.173+0000 7fca7ae43640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:17.174 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.173+0000 7fca7ae43640 4 set_mon_vals no callback set 2024-03-25T18:08:17.175 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.173+0000 7fca737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:17.175 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.173+0000 7fca737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:17.175 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.173+0000 7fca737fe640 10 monclient: dump: 2024-03-25T18:08:17.175 INFO:teuthology.orchestra.run.smithi156.stderr:epoch 1 2024-03-25T18:08:17.175 INFO:teuthology.orchestra.run.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:17.175 INFO:teuthology.orchestra.run.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:17.175 INFO:teuthology.orchestra.run.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:17.175 INFO:teuthology.orchestra.run.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:17.175 INFO:teuthology.orchestra.run.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:17.175 INFO:teuthology.orchestra.run.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:17.175 INFO:teuthology.orchestra.run.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:17.175 INFO:teuthology.orchestra.run.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:17.175 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-25T18:08:17.178 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.174+0000 7fca7ae43640 10 monclient: shutdown 2024-03-25T18:08:17.178 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.174+0000 7fca7ae43640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:17.178 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.174+0000 7fca7ae43640 1 -- 172.21.15.156:0/342947227 >> v1:172.21.15.156:6789/0 conn(0x7fca740a9920 legacy=0x7fca580056b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:17.179 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.174+0000 7fca7ae43640 1 -- 172.21.15.156:0/342947227 shutdown_connections 2024-03-25T18:08:17.179 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.174+0000 7fca7ae43640 1 -- 172.21.15.156:0/342947227 >> 172.21.15.156:0/342947227 conn(0x7fca740b7f40 msgr2=0x7fca740b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:17.179 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.175+0000 7fca7ae43640 1 -- 172.21.15.156:0/342947227 shutdown_connections 2024-03-25T18:08:17.179 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.175+0000 7fca7ae43640 1 -- 172.21.15.156:0/342947227 wait complete. 2024-03-25T18:08:17.179 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.175+0000 7fca7ae43640 10 monclient: build_initial_monmap 2024-03-25T18:08:17.179 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.175+0000 7fca7ae43640 10 monclient: monmap: 2024-03-25T18:08:17.179 INFO:teuthology.orchestra.run.smithi156.stderr:epoch 0 2024-03-25T18:08:17.179 INFO:teuthology.orchestra.run.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:17.179 INFO:teuthology.orchestra.run.smithi156.stderr:last_changed 0.000000 2024-03-25T18:08:17.179 INFO:teuthology.orchestra.run.smithi156.stderr:created 0.000000 2024-03-25T18:08:17.179 INFO:teuthology.orchestra.run.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:17.179 INFO:teuthology.orchestra.run.smithi156.stderr:election_strategy: 1 2024-03-25T18:08:17.179 INFO:teuthology.orchestra.run.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:17.179 INFO:teuthology.orchestra.run.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:08:17.179 INFO:teuthology.orchestra.run.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:08:17.180 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-25T18:08:17.180 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.175+0000 7fca7ae43640 1 librados: starting msgr at 2024-03-25T18:08:17.180 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.175+0000 7fca7ae43640 1 librados: starting objecter 2024-03-25T18:08:17.180 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.176+0000 7fca7ae43640 1 Processor -- start 2024-03-25T18:08:17.180 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.176+0000 7fca7ae43640 1 -- start start 2024-03-25T18:08:17.180 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.176+0000 7fca7ae43640 1 librados: setting wanted keys 2024-03-25T18:08:17.180 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.176+0000 7fca7ae43640 1 librados: calling monclient init 2024-03-25T18:08:17.180 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.176+0000 7fca7ae43640 10 monclient: init 2024-03-25T18:08:17.180 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.176+0000 7fca7ae43640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:17.180 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.176+0000 7fca7ae43640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:08:17.180 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.176+0000 7fca7ae43640 10 monclient(hunting): picked mon.noname-a con 0x7fca5800fdd0 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:17.180 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.176+0000 7fca7ae43640 10 monclient(hunting): picked mon.noname-b con 0x7fca580120a0 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:17.180 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.176+0000 7fca7ae43640 10 monclient(hunting): picked mon.noname-c con 0x7fca740a9920 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:17.180 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.176+0000 7fca7ae43640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca741d76e0 con 0x7fca5800fdd0 2024-03-25T18:08:17.180 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.176+0000 7fca7ae43640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca740ad4e0 con 0x7fca580120a0 2024-03-25T18:08:17.180 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.176+0000 7fca7ae43640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca741d8280 con 0x7fca740a9920 2024-03-25T18:08:17.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.176+0000 7fca7ae43640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:17.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.176+0000 7fca7ae43640 10 monclient(hunting): authenticate will time out at 6978.487695s 2024-03-25T18:08:17.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.177+0000 7fca793e0640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7fca740a9920 0x7fca741d7b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:37764/0 (socket says 172.21.15.156:37764) 2024-03-25T18:08:17.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.177+0000 7fca793e0640 1 -- 172.21.15.156:0/901076124 learned_addr learned my addr 172.21.15.156:0/901076124 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:08:17.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.177+0000 7fca717fa640 10 client.?.objecter ms_handle_connect 0x7fca740a9920 2024-03-25T18:08:17.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.177+0000 7fca717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:17.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.177+0000 7fca717fa640 10 client.?.objecter ms_handle_connect 0x7fca580120a0 2024-03-25T18:08:17.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.177+0000 7fca717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:17.216 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.215+0000 7fca717fa640 10 client.?.objecter ms_handle_connect 0x7fca5800fdd0 2024-03-25T18:08:17.216 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.215+0000 7fca717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:17.228 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.227+0000 7fca717fa640 1 -- 172.21.15.156:0/901076124 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 215872131 0 0) 0x7fca741d76e0 con 0x7fca5800fdd0 2024-03-25T18:08:17.228 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.227+0000 7fca717fa640 10 monclient(hunting): _negotiate 2024-03-25T18:08:17.228 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.227+0000 7fca717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:17.228 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.227+0000 7fca717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:17.228 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.227+0000 7fca717fa640 10 monclient(hunting): my global_id is 4107 2024-03-25T18:08:17.228 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.227+0000 7fca717fa640 1 -- 172.21.15.156:0/901076124 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca580034d0 con 0x7fca5800fdd0 2024-03-25T18:08:17.237 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.237+0000 7fca717fa640 1 -- 172.21.15.156:0/901076124 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789931094 0 0) 0x7fca740ad4e0 con 0x7fca580120a0 2024-03-25T18:08:17.237 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.237+0000 7fca717fa640 10 monclient(hunting): _negotiate 2024-03-25T18:08:17.237 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.237+0000 7fca717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:17.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.237+0000 7fca717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:17.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.237+0000 7fca717fa640 10 monclient(hunting): my global_id is 4108 2024-03-25T18:08:17.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.237+0000 7fca717fa640 1 -- 172.21.15.156:0/901076124 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca741d76e0 con 0x7fca580120a0 2024-03-25T18:08:17.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.240+0000 7fca717fa640 1 -- 172.21.15.156:0/901076124 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1376644998 0 0) 0x7fca741d8280 con 0x7fca740a9920 2024-03-25T18:08:17.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.240+0000 7fca717fa640 10 monclient(hunting): _negotiate 2024-03-25T18:08:17.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.240+0000 7fca717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:17.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.240+0000 7fca717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:17.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.240+0000 7fca717fa640 10 monclient(hunting): my global_id is 4109 2024-03-25T18:08:17.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.240+0000 7fca717fa640 1 -- 172.21.15.156:0/901076124 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca740ad4e0 con 0x7fca740a9920 2024-03-25T18:08:17.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.251+0000 7fca717fa640 1 -- 172.21.15.156:0/901076124 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 137665265 0 0) 0x7fca580034d0 con 0x7fca5800fdd0 2024-03-25T18:08:17.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.251+0000 7fca717fa640 1 -- 172.21.15.156:0/901076124 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca741d8280 con 0x7fca5800fdd0 2024-03-25T18:08:17.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.251+0000 7fca717fa640 1 -- 172.21.15.156:0/901076124 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fca68004230 con 0x7fca5800fdd0 2024-03-25T18:08:17.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.251+0000 7fca717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:17.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.251+0000 7fca717fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:08:17.252 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.251+0000 7fca717fa640 10 monclient(hunting): dump: 2024-03-25T18:08:17.252 INFO:teuthology.orchestra.run.smithi156.stderr:epoch 1 2024-03-25T18:08:17.252 INFO:teuthology.orchestra.run.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:17.252 INFO:teuthology.orchestra.run.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:17.252 INFO:teuthology.orchestra.run.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:17.252 INFO:teuthology.orchestra.run.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:17.252 INFO:teuthology.orchestra.run.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:17.252 INFO:teuthology.orchestra.run.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:17.252 INFO:teuthology.orchestra.run.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:17.252 INFO:teuthology.orchestra.run.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:17.252 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-25T18:08:17.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.251+0000 7fca717fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:17.264 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.263+0000 7fca717fa640 1 -- 172.21.15.156:0/901076124 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1780219611 0 0) 0x7fca741d8280 con 0x7fca5800fdd0 2024-03-25T18:08:17.264 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.263+0000 7fca717fa640 1 -- 172.21.15.156:0/901076124 >> v1:172.21.15.156:6790/0 conn(0x7fca740a9920 legacy=0x7fca741d7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:17.264 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.263+0000 7fca717fa640 1 -- 172.21.15.156:0/901076124 >> v1:172.21.15.194:6789/0 conn(0x7fca580120a0 legacy=0x7fca74142e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:17.264 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.263+0000 7fca717fa640 10 monclient: _finish_hunting 0 2024-03-25T18:08:17.264 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.263+0000 7fca717fa640 1 monclient: found mon.a 2024-03-25T18:08:17.264 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.263+0000 7fca717fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:17.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.263+0000 7fca717fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:08:17.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.263+0000 7fca717fa640 1 -- 172.21.15.156:0/901076124 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca741d8410 con 0x7fca5800fdd0 2024-03-25T18:08:17.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.263+0000 7fca717fa640 10 monclient: _finish_auth 0 2024-03-25T18:08:17.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.263+0000 7fca717fa640 10 monclient: _check_auth_tickets 2024-03-25T18:08:17.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.263+0000 7fca717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:17.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.263+0000 7fca7ae43640 5 monclient: authenticate success, global_id 4107 2024-03-25T18:08:17.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.263+0000 7fca7ae43640 10 monclient: _renew_subs 2024-03-25T18:08:17.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.263+0000 7fca7ae43640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:08:17.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.263+0000 7fca7ae43640 1 -- 172.21.15.156:0/901076124 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca741401f0 con 0x7fca5800fdd0 2024-03-25T18:08:17.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.263+0000 7fca7ae43640 10 client.4107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:17.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.263+0000 7fca7ae43640 10 monclient: _renew_subs 2024-03-25T18:08:17.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.263+0000 7fca7ae43640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:08:17.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.263+0000 7fca7ae43640 1 -- 172.21.15.156:0/901076124 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca741404a0 con 0x7fca5800fdd0 2024-03-25T18:08:17.265 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.263+0000 7fca7ae43640 1 librados: init done 2024-03-25T18:08:17.266 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.264+0000 7fca7ae43640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:17.266 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.264+0000 7fca7ae43640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:17.266 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.264+0000 7fca7ae43640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:08:17.266 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.264+0000 7fca7ae43640 1 -- 172.21.15.156:0/901076124 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca7404f730 con 0x7fca5800fdd0 2024-03-25T18:08:17.491 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.490+0000 7fca717fa640 1 -- 172.21.15.156:0/901076124 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca68004640 con 0x7fca5800fdd0 2024-03-25T18:08:17.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.490+0000 7fca717fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:17.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.490+0000 7fca717fa640 1 -- 172.21.15.156:0/901076124 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fca68004830 con 0x7fca5800fdd0 2024-03-25T18:08:17.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.490+0000 7fca717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:17.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.490+0000 7fca717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:17.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.490+0000 7fca717fa640 10 monclient: dump: 2024-03-25T18:08:17.492 INFO:teuthology.orchestra.run.smithi156.stderr:epoch 1 2024-03-25T18:08:17.492 INFO:teuthology.orchestra.run.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:17.492 INFO:teuthology.orchestra.run.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:17.492 INFO:teuthology.orchestra.run.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:17.492 INFO:teuthology.orchestra.run.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:17.492 INFO:teuthology.orchestra.run.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:17.493 INFO:teuthology.orchestra.run.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:17.493 INFO:teuthology.orchestra.run.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:17.493 INFO:teuthology.orchestra.run.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:17.493 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-25T18:08:17.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.490+0000 7fca717fa640 1 -- 172.21.15.156:0/901076124 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fca68004de0 con 0x7fca5800fdd0 2024-03-25T18:08:17.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.490+0000 7fca72ffd640 4 set_mon_vals no callback set 2024-03-25T18:08:17.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.490+0000 7fca717fa640 1 -- 172.21.15.156:0/901076124 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3972386208 0 0) 0x7fca68006430 con 0x7fca5800fdd0 2024-03-25T18:08:17.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.490+0000 7fca717fa640 10 client.4107.objecter ms_dispatch 0x7fca74000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T18:08:17.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.490+0000 7fca717fa640 3 client.4107.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T18:08:17.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.490+0000 7fca717fa640 3 client.4107.objecter handle_osd_map decoding full epoch 1 2024-03-25T18:08:17.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.490+0000 7fca717fa640 20 client.4107.objecter dump_active .. 0 homeless 2024-03-25T18:08:17.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.490+0000 7fca717fa640 1 -- 172.21.15.156:0/901076124 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fca741d8410 con 0x7fca5800fdd0 2024-03-25T18:08:17.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.490+0000 7fca717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:17.494 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.490+0000 7fca717fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:17.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.567+0000 7fca7ae43640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:08:17.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.567+0000 7fca7ae43640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:08:17.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.567+0000 7fca7ae43640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:08:17.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.567+0000 7fca7ae43640 1 -- 172.21.15.156:0/901076124 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fca740b65b0 con 0x7fca5800fdd0 2024-03-25T18:08:17.812 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.812+0000 7fca717fa640 1 -- 172.21.15.156:0/901076124 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fca68005090 con 0x7fca5800fdd0 2024-03-25T18:08:17.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.812+0000 7fca717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:08:17.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.812+0000 7fca717fa640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T18:08:17.813 INFO:teuthology.orchestra.run.smithi156.stderr:adjusted tunables profile to default 2024-03-25T18:08:17.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.812+0000 7fca7ae43640 10 librados: watch_flush enter 2024-03-25T18:08:17.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.812+0000 7fca7ae43640 10 librados: watch_flush exit 2024-03-25T18:08:17.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.812+0000 7fca7ae43640 20 client.4107.objecter shutdown clearing up homeless session... 2024-03-25T18:08:17.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.812+0000 7fca7ae43640 10 client.4107.objecter successfully canceled tick 2024-03-25T18:08:17.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.813+0000 7fca7ae43640 10 monclient: shutdown 2024-03-25T18:08:17.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.813+0000 7fca7ae43640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:17.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.813+0000 7fca7ae43640 1 -- 172.21.15.156:0/901076124 >> v1:172.21.15.156:6789/0 conn(0x7fca5800fdd0 legacy=0x7fca740abe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:17.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.813+0000 7fca7ae43640 1 -- 172.21.15.156:0/901076124 shutdown_connections 2024-03-25T18:08:17.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.813+0000 7fca7ae43640 1 -- 172.21.15.156:0/901076124 >> 172.21.15.156:0/901076124 conn(0x7fca740b7f40 msgr2=0x7fca740b2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:17.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.813+0000 7fca7ae43640 1 -- 172.21.15.156:0/901076124 shutdown_connections 2024-03-25T18:08:17.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.813+0000 7fca7ae43640 1 -- 172.21.15.156:0/901076124 wait complete. 2024-03-25T18:08:17.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:17.813+0000 7fca7ae43640 1 librados: shutdown 2024-03-25T18:08:17.825 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T18:08:17.825 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T18:08:17.826 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-25T18:08:17.826 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T18:08:17.854 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-25T18:08:17.855 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T18:08:17.873 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:08:17.871+0000 7f0803836040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:08:17.922 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-25T18:08:17.923 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T18:08:17.927 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:08:17.925+0000 7f0803836040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:08:17.941 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-25T18:08:17.941+0000 7f776ee13040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:08:17.970 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-25T18:08:17.970 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T18:08:18.004 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-25T18:08:18.003+0000 7f776ee13040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:08:18.032 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-25T18:08:18.032 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T18:08:18.061 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-25T18:08:18.061 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T18:08:18.126 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-25T18:08:18.126 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T18:08:18.193 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-25T18:08:18.193 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T18:08:18.260 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new c34cf722-8f77-4343-a481-69e0c03a1ecd 0 2024-03-25T18:08:18.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.399+0000 7f0588e0f640 10 monclient: get_monmap_and_config 2024-03-25T18:08:18.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.399+0000 7f0588e0f640 10 monclient: build_initial_monmap 2024-03-25T18:08:18.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.399+0000 7f0588e0f640 10 monclient: monmap: 2024-03-25T18:08:18.401 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-25T18:08:18.402 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:18.402 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:08:18.401651+0000 2024-03-25T18:08:18.402 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:08:18.401651+0000 2024-03-25T18:08:18.402 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:18.402 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:18.402 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:18.402 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:08:18.402 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:08:18.402 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:18.402 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.400+0000 7f0588e0f640 1 Processor -- start 2024-03-25T18:08:18.402 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.400+0000 7f0588e0f640 1 -- start start 2024-03-25T18:08:18.402 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.400+0000 7f0588e0f640 10 monclient: init 2024-03-25T18:08:18.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.401+0000 7f0588e0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:18.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.401+0000 7f0588e0f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:08:18.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.401+0000 7f0588e0f640 10 monclient(hunting): picked mon.noname-c con 0x7f05840b4700 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:18.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.401+0000 7f0588e0f640 10 monclient(hunting): picked mon.noname-a con 0x7f0584142a20 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:18.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.401+0000 7f0588e0f640 10 monclient(hunting): picked mon.noname-b con 0x7f05840a96b0 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:18.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.401+0000 7f0588e0f640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05840a9d60 con 0x7f0584142a20 2024-03-25T18:08:18.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.401+0000 7f0588e0f640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058413ff10 con 0x7f05840b4700 2024-03-25T18:08:18.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.401+0000 7f0588e0f640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05840aa1f0 con 0x7f05840a96b0 2024-03-25T18:08:18.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.401+0000 7f0588e0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:18.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.401+0000 7f0588e0f640 10 monclient(hunting): authenticate will time out at 6968.683850s 2024-03-25T18:08:18.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.401+0000 7f058259c640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f05840b4700 0x7f0584142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:34196/0 (socket says 172.21.15.194:34196) 2024-03-25T18:08:18.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.401+0000 7f058259c640 1 -- 172.21.15.194:0/213614914 learned_addr learned my addr 172.21.15.194:0/213614914 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-25T18:08:18.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.404+0000 7f058159a640 1 -- 172.21.15.194:0/213614914 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2933720981 0 0) 0x7f058413ff10 con 0x7f05840b4700 2024-03-25T18:08:18.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.404+0000 7f058159a640 10 monclient(hunting): _negotiate 2024-03-25T18:08:18.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.404+0000 7f058159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:18.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.404+0000 7f058159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:18.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.404+0000 7f058159a640 10 monclient(hunting): my global_id is 4117 2024-03-25T18:08:18.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.404+0000 7f058159a640 1 -- 172.21.15.194:0/213614914 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f056c003520 con 0x7f05840b4700 2024-03-25T18:08:18.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.404+0000 7f058159a640 1 -- 172.21.15.194:0/213614914 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 207803753 0 0) 0x7f05840a9d60 con 0x7f0584142a20 2024-03-25T18:08:18.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.404+0000 7f058159a640 10 monclient(hunting): _negotiate 2024-03-25T18:08:18.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.404+0000 7f058159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:18.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.404+0000 7f058159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:18.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.404+0000 7f058159a640 10 monclient(hunting): my global_id is 4116 2024-03-25T18:08:18.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.404+0000 7f058159a640 1 -- 172.21.15.194:0/213614914 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f058413ff10 con 0x7f0584142a20 2024-03-25T18:08:18.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.404+0000 7f058159a640 1 -- 172.21.15.194:0/213614914 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126445909 0 0) 0x7f05840aa1f0 con 0x7f05840a96b0 2024-03-25T18:08:18.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.404+0000 7f058159a640 10 monclient(hunting): _negotiate 2024-03-25T18:08:18.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.404+0000 7f058159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:18.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.404+0000 7f058159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:18.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.404+0000 7f058159a640 10 monclient(hunting): my global_id is 4118 2024-03-25T18:08:18.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.404+0000 7f058159a640 1 -- 172.21.15.194:0/213614914 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05840a9d60 con 0x7f05840a96b0 2024-03-25T18:08:18.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.407+0000 7f058159a640 1 -- 172.21.15.194:0/213614914 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2932394695 0 0) 0x7f056c003520 con 0x7f05840b4700 2024-03-25T18:08:18.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.407+0000 7f058159a640 1 -- 172.21.15.194:0/213614914 >> v1:172.21.15.156:6790/0 conn(0x7f05840a96b0 legacy=0x7f058413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:18.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.407+0000 7f058159a640 1 -- 172.21.15.194:0/213614914 >> v1:172.21.15.156:6789/0 conn(0x7f0584142a20 legacy=0x7f058413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:18.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.407+0000 7f058159a640 10 monclient: _finish_hunting 0 2024-03-25T18:08:18.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.407+0000 7f058159a640 1 monclient: found mon.noname-c 2024-03-25T18:08:18.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.407+0000 7f058159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:18.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.407+0000 7f058159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:08:18.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.407+0000 7f058159a640 1 -- 172.21.15.194:0/213614914 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05840b4ae0 con 0x7f05840b4700 2024-03-25T18:08:18.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.407+0000 7f058159a640 10 monclient: _finish_auth 0 2024-03-25T18:08:18.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.407+0000 7f058159a640 10 monclient: _check_auth_tickets 2024-03-25T18:08:18.410 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.407+0000 7f058159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:18.410 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.407+0000 7f058159a640 1 -- 172.21.15.194:0/213614914 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f0574001690 con 0x7f05840b4700 2024-03-25T18:08:18.410 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.407+0000 7f058159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:18.410 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.407+0000 7f058159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:18.410 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.407+0000 7f058159a640 10 monclient: dump: 2024-03-25T18:08:18.410 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:18.410 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:18.410 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:18.410 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:18.410 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:18.410 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:18.410 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:18.410 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:18.410 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:18.410 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:18.411 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.407+0000 7f0588e0f640 5 monclient: authenticate success, global_id 4117 2024-03-25T18:08:18.411 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.407+0000 7f0588e0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:18.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.433+0000 7f058159a640 1 -- 172.21.15.194:0/213614914 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05740017f0 con 0x7f05840b4700 2024-03-25T18:08:18.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.433+0000 7f058159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:18.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.433+0000 7f058159a640 1 -- 172.21.15.194:0/213614914 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f0574001c00 con 0x7f05840b4700 2024-03-25T18:08:18.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.433+0000 7f058159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:18.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.433+0000 7f058159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:18.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.433+0000 7f058159a640 10 monclient: dump: 2024-03-25T18:08:18.435 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:18.435 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:18.436 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:18.436 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:18.436 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:18.436 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:18.436 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:18.436 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:18.436 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:18.436 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:18.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.433+0000 7f0588e0f640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:18.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.433+0000 7f0588e0f640 4 set_mon_vals no callback set 2024-03-25T18:08:18.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.434+0000 7f0588e0f640 10 monclient: shutdown 2024-03-25T18:08:18.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.434+0000 7f0588e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:18.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.434+0000 7f0588e0f640 1 -- 172.21.15.194:0/213614914 >> v1:172.21.15.194:6789/0 conn(0x7f05840b4700 legacy=0x7f0584142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:18.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.434+0000 7f0588e0f640 1 -- 172.21.15.194:0/213614914 shutdown_connections 2024-03-25T18:08:18.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.434+0000 7f0588e0f640 1 -- 172.21.15.194:0/213614914 >> 172.21.15.194:0/213614914 conn(0x7f05840668e0 msgr2=0x7f0584066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:18.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.434+0000 7f0588e0f640 1 -- 172.21.15.194:0/213614914 shutdown_connections 2024-03-25T18:08:18.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.434+0000 7f0588e0f640 1 -- 172.21.15.194:0/213614914 wait complete. 2024-03-25T18:08:18.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.435+0000 7f0588e0f640 10 monclient: build_initial_monmap 2024-03-25T18:08:18.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.435+0000 7f0588e0f640 10 monclient: monmap: 2024-03-25T18:08:18.439 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-25T18:08:18.439 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:18.439 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 0.000000 2024-03-25T18:08:18.439 INFO:teuthology.orchestra.run.smithi194.stderr:created 0.000000 2024-03-25T18:08:18.440 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:18.440 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-25T18:08:18.440 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:18.440 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:08:18.440 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:08:18.440 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:18.440 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.435+0000 7f0588e0f640 1 librados: starting msgr at 2024-03-25T18:08:18.440 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.435+0000 7f0588e0f640 1 librados: starting objecter 2024-03-25T18:08:18.440 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.435+0000 7f0588e0f640 1 Processor -- start 2024-03-25T18:08:18.440 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.435+0000 7f0588e0f640 1 -- start start 2024-03-25T18:08:18.440 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.435+0000 7f0588e0f640 1 librados: setting wanted keys 2024-03-25T18:08:18.440 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.435+0000 7f0588e0f640 1 librados: calling monclient init 2024-03-25T18:08:18.440 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.435+0000 7f0588e0f640 10 monclient: init 2024-03-25T18:08:18.440 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.436+0000 7f0588e0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:18.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.436+0000 7f0588e0f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:08:18.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.436+0000 7f0588e0f640 10 monclient(hunting): picked mon.noname-a con 0x7f05840a96b0 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:18.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.436+0000 7f0588e0f640 10 monclient(hunting): picked mon.noname-b con 0x7f05840b4700 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:18.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.436+0000 7f0588e0f640 10 monclient(hunting): picked mon.noname-c con 0x7f0584142a20 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:18.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.436+0000 7f0588e0f640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05841d4fe0 con 0x7f05840a96b0 2024-03-25T18:08:18.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.436+0000 7f0588e0f640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05841d7760 con 0x7f05840b4700 2024-03-25T18:08:18.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.436+0000 7f0588e0f640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05841d73f0 con 0x7f0584142a20 2024-03-25T18:08:18.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.436+0000 7f0588e0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:18.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.436+0000 7f0588e0f640 10 monclient(hunting): authenticate will time out at 6968.719001s 2024-03-25T18:08:18.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.436+0000 7f0581d9b640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f05840b4700 0x7f05840ac2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:34200/0 (socket says 172.21.15.194:34200) 2024-03-25T18:08:18.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.436+0000 7f0581d9b640 1 -- 172.21.15.194:0/205654046 learned_addr learned my addr 172.21.15.194:0/205654046 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-25T18:08:18.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.437+0000 7f0572ffd640 10 client.?.objecter ms_handle_connect 0x7f05840b4700 2024-03-25T18:08:18.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.437+0000 7f0572ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:18.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.437+0000 7f0572ffd640 10 client.?.objecter ms_handle_connect 0x7f0584142a20 2024-03-25T18:08:18.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.437+0000 7f0572ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:18.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.437+0000 7f0572ffd640 10 client.?.objecter ms_handle_connect 0x7f05840a96b0 2024-03-25T18:08:18.442 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.437+0000 7f0572ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:18.442 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.439+0000 7f0572ffd640 1 -- 172.21.15.194:0/205654046 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2661743958 0 0) 0x7f05841d7760 con 0x7f05840b4700 2024-03-25T18:08:18.443 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.439+0000 7f0572ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:18.443 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.439+0000 7f0572ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:18.443 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.439+0000 7f0572ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:18.443 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.439+0000 7f0572ffd640 10 monclient(hunting): my global_id is 4120 2024-03-25T18:08:18.443 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.439+0000 7f0572ffd640 1 -- 172.21.15.194:0/205654046 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0564004eb0 con 0x7f05840b4700 2024-03-25T18:08:18.443 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.439+0000 7f0572ffd640 1 -- 172.21.15.194:0/205654046 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4250302563 0 0) 0x7f05841d73f0 con 0x7f0584142a20 2024-03-25T18:08:18.443 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.439+0000 7f0572ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:18.443 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.439+0000 7f0572ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:18.443 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.439+0000 7f0572ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:18.443 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.439+0000 7f0572ffd640 10 monclient(hunting): my global_id is 4121 2024-03-25T18:08:18.443 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.439+0000 7f0572ffd640 1 -- 172.21.15.194:0/205654046 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05841d7760 con 0x7f0584142a20 2024-03-25T18:08:18.443 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.440+0000 7f0572ffd640 1 -- 172.21.15.194:0/205654046 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1782403816 0 0) 0x7f05841d4fe0 con 0x7f05840a96b0 2024-03-25T18:08:18.443 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.440+0000 7f0572ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:18.443 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.440+0000 7f0572ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:18.443 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.440+0000 7f0572ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:18.443 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.440+0000 7f0572ffd640 10 monclient(hunting): my global_id is 4119 2024-03-25T18:08:18.444 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.440+0000 7f0572ffd640 1 -- 172.21.15.194:0/205654046 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05841d73f0 con 0x7f05840a96b0 2024-03-25T18:08:18.445 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.443+0000 7f0572ffd640 1 -- 172.21.15.194:0/205654046 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3612193724 0 0) 0x7f0564004eb0 con 0x7f05840b4700 2024-03-25T18:08:18.445 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.443+0000 7f0572ffd640 1 -- 172.21.15.194:0/205654046 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05841d4fe0 con 0x7f05840b4700 2024-03-25T18:08:18.446 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.443+0000 7f0572ffd640 1 -- 172.21.15.194:0/205654046 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f0578001a30 con 0x7f05840b4700 2024-03-25T18:08:18.446 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.443+0000 7f0572ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:18.446 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.443+0000 7f0572ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:18.446 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.443+0000 7f0572ffd640 10 monclient(hunting): dump: 2024-03-25T18:08:18.446 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:18.446 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:18.446 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:18.446 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:18.446 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:18.446 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:18.446 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:18.446 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:18.446 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:18.447 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:18.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.443+0000 7f0572ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:18.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.444+0000 7f0572ffd640 1 -- 172.21.15.194:0/205654046 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 951142620 0 0) 0x7f05841d7760 con 0x7f0584142a20 2024-03-25T18:08:18.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.444+0000 7f0572ffd640 1 -- 172.21.15.194:0/205654046 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0564004eb0 con 0x7f0584142a20 2024-03-25T18:08:18.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.444+0000 7f0572ffd640 1 -- 172.21.15.194:0/205654046 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f057c001b50 con 0x7f0584142a20 2024-03-25T18:08:18.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.444+0000 7f0572ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:18.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.444+0000 7f0572ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:18.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.444+0000 7f0572ffd640 10 monclient(hunting): dump: 2024-03-25T18:08:18.447 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:18.447 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:18.447 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:18.447 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:18.447 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:18.447 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:18.447 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:18.447 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:18.448 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:18.448 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:18.449 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.447+0000 7f0572ffd640 1 -- 172.21.15.194:0/205654046 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2799240277 0 0) 0x7f05841d73f0 con 0x7f05840a96b0 2024-03-25T18:08:18.449 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.447+0000 7f0572ffd640 1 -- 172.21.15.194:0/205654046 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05841d7760 con 0x7f05840a96b0 2024-03-25T18:08:18.449 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.447+0000 7f0572ffd640 1 -- 172.21.15.194:0/205654046 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f0574001b00 con 0x7f05840a96b0 2024-03-25T18:08:18.449 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.447+0000 7f0572ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:18.449 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.447+0000 7f0572ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:18.449 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.447+0000 7f0572ffd640 10 monclient(hunting): dump: 2024-03-25T18:08:18.449 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:18.449 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:18.449 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:18.450 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:18.450 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:18.450 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:18.450 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:18.450 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:18.450 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:18.450 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:18.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.448+0000 7f0572ffd640 1 -- 172.21.15.194:0/205654046 <== mon.2 v1:172.21.15.156:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2952633108 0 0) 0x7f0564004eb0 con 0x7f0584142a20 2024-03-25T18:08:18.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.448+0000 7f0572ffd640 1 -- 172.21.15.194:0/205654046 >> v1:172.21.15.194:6789/0 conn(0x7f05840b4700 legacy=0x7f05840ac2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:18.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.448+0000 7f0572ffd640 1 -- 172.21.15.194:0/205654046 >> v1:172.21.15.156:6789/0 conn(0x7f05840a96b0 legacy=0x7f05841d7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:18.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.448+0000 7f0572ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:08:18.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.448+0000 7f0572ffd640 1 monclient: found mon.c 2024-03-25T18:08:18.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.448+0000 7f0572ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:18.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.448+0000 7f0572ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:08:18.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.448+0000 7f0572ffd640 1 -- 172.21.15.194:0/205654046 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05840aca00 con 0x7f0584142a20 2024-03-25T18:08:18.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.448+0000 7f0572ffd640 10 monclient: _finish_auth 0 2024-03-25T18:08:18.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.448+0000 7f0572ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:08:18.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.448+0000 7f0572ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:18.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.448+0000 7f0588e0f640 5 monclient: authenticate success, global_id 4121 2024-03-25T18:08:18.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.448+0000 7f0588e0f640 10 monclient: _renew_subs 2024-03-25T18:08:18.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.448+0000 7f0588e0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:08:18.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.448+0000 7f0588e0f640 1 -- 172.21.15.194:0/205654046 --> v1:172.21.15.156:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05840acc50 con 0x7f0584142a20 2024-03-25T18:08:18.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.448+0000 7f0588e0f640 10 client.4121.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:18.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.448+0000 7f0588e0f640 10 monclient: _renew_subs 2024-03-25T18:08:18.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.448+0000 7f0588e0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:08:18.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.448+0000 7f0588e0f640 1 -- 172.21.15.194:0/205654046 --> v1:172.21.15.156:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f05841e3520 con 0x7f0584142a20 2024-03-25T18:08:18.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.448+0000 7f0588e0f640 1 librados: init done 2024-03-25T18:08:18.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.449+0000 7f0588e0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:18.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.449+0000 7f0588e0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:18.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.449+0000 7f0588e0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:08:18.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.449+0000 7f0588e0f640 1 -- 172.21.15.194:0/205654046 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05841d7580 con 0x7f0584142a20 2024-03-25T18:08:18.455 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.453+0000 7f0572ffd640 1 -- 172.21.15.194:0/205654046 <== mon.2 v1:172.21.15.156:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f057c001fe0 con 0x7f0584142a20 2024-03-25T18:08:18.455 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.453+0000 7f0572ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:18.455 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.453+0000 7f0572ffd640 1 -- 172.21.15.194:0/205654046 <== mon.2 v1:172.21.15.156:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f057c003b10 con 0x7f0584142a20 2024-03-25T18:08:18.455 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.453+0000 7f0572ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:18.455 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.453+0000 7f0572ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:18.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.453+0000 7f0572ffd640 10 monclient: dump: 2024-03-25T18:08:18.456 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:18.456 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:18.456 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:18.456 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:18.456 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:18.456 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:18.456 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:18.456 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:18.456 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:18.456 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:18.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.453+0000 7f0572ffd640 1 -- 172.21.15.194:0/205654046 <== mon.2 v1:172.21.15.156:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f057c004000 con 0x7f0584142a20 2024-03-25T18:08:18.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.453+0000 7f0580d99640 4 set_mon_vals no callback set 2024-03-25T18:08:18.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.453+0000 7f0572ffd640 1 -- 172.21.15.194:0/205654046 <== mon.2 v1:172.21.15.156:6790/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 2371979830 0 0) 0x7f057c005460 con 0x7f0584142a20 2024-03-25T18:08:18.457 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.453+0000 7f0572ffd640 10 client.4121.objecter ms_dispatch 0x7f0584000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T18:08:18.457 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.453+0000 7f0572ffd640 3 client.4121.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T18:08:18.457 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.453+0000 7f0572ffd640 3 client.4121.objecter handle_osd_map decoding full epoch 2 2024-03-25T18:08:18.457 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.453+0000 7f0572ffd640 20 client.4121.objecter dump_active .. 0 homeless 2024-03-25T18:08:18.491 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:08:18.490+0000 7f0803836040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:08:18.545 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:08:18.544+0000 7f0803836040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:08:18.563 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-25T18:08:18.563+0000 7f776ee13040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:08:18.589 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:08:18.588+0000 7f0803836040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:08:18.617 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-25T18:08:18.617+0000 7f776ee13040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:08:18.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.619+0000 7f0572ffd640 1 -- 172.21.15.194:0/205654046 <== mon.2 v1:172.21.15.156:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f057c005840 con 0x7f0584142a20 2024-03-25T18:08:18.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.619+0000 7f0572ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:18.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.619+0000 7f0572ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:18.644 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:08:18.642+0000 7f0803836040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:08:18.662 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-25T18:08:18.662+0000 7f776ee13040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:08:18.687 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:08:18.686+0000 7f0803836040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:08:18.693 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.691+0000 7f0588e0f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c34cf722-8f77-4343-a481-69e0c03a1ecd", "id": 0}] 2024-03-25T18:08:18.693 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.691+0000 7f0588e0f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c34cf722-8f77-4343-a481-69e0c03a1ecd", "id": 0}] 2024-03-25T18:08:18.693 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.691+0000 7f0588e0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:08:18.693 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.691+0000 7f0588e0f640 1 -- 172.21.15.194:0/205654046 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "osd new", "uuid": "c34cf722-8f77-4343-a481-69e0c03a1ecd", "id": 0} v 0) v1 -- 0x7f05840ae130 con 0x7f0584142a20 2024-03-25T18:08:18.717 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-25T18:08:18.716+0000 7f776ee13040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:08:18.779 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-25T18:08:18.778+0000 7f776ee13040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:08:18.801 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:08:18.800+0000 7f0803836040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:08:18.872 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:08:18.871+0000 7f0803836040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:08:18.915 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-25T18:08:18.915+0000 7f776ee13040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:08:18.919 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:08:18.917+0000 7f0803836040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:08:18.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.922+0000 7f0572ffd640 1 -- 172.21.15.194:0/205654046 <== mon.2 v1:172.21.15.156:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c34cf722-8f77-4343-a481-69e0c03a1ecd", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 2702520900 0 732707515) 0x7f057c005ad0 con 0x7f0584142a20 2024-03-25T18:08:18.924 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.922+0000 7f0572ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c34cf722-8f77-4343-a481-69e0c03a1ecd", "id": 0}] 2024-03-25T18:08:18.924 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.922+0000 7f0572ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:18.924 INFO:teuthology.orchestra.run.smithi194.stdout:0 2024-03-25T18:08:18.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.923+0000 7f0588e0f640 10 librados: watch_flush enter 2024-03-25T18:08:18.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.923+0000 7f0588e0f640 10 librados: watch_flush exit 2024-03-25T18:08:18.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.923+0000 7f0588e0f640 20 client.4121.objecter shutdown clearing up homeless session... 2024-03-25T18:08:18.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.923+0000 7f0588e0f640 10 client.4121.objecter successfully canceled tick 2024-03-25T18:08:18.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.923+0000 7f0588e0f640 10 monclient: shutdown 2024-03-25T18:08:18.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.923+0000 7f0588e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:18.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.923+0000 7f0588e0f640 1 -- 172.21.15.194:0/205654046 >> v1:172.21.15.156:6790/0 conn(0x7f0584142a20 legacy=0x7f05841e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:18.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.923+0000 7f0588e0f640 1 -- 172.21.15.194:0/205654046 shutdown_connections 2024-03-25T18:08:18.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.923+0000 7f0588e0f640 1 -- 172.21.15.194:0/205654046 >> 172.21.15.194:0/205654046 conn(0x7f05840668e0 msgr2=0x7f05840b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:18.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.923+0000 7f0588e0f640 1 -- 172.21.15.194:0/205654046 shutdown_connections 2024-03-25T18:08:18.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.923+0000 7f0588e0f640 1 -- 172.21.15.194:0/205654046 wait complete. 2024-03-25T18:08:18.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:18.923+0000 7f0588e0f640 1 librados: shutdown 2024-03-25T18:08:18.935 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb 1 2024-03-25T18:08:18.977 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:08:18.976+0000 7f0803836040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:08:18.997 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-25T18:08:18.997+0000 7f776ee13040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:08:19.028 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:08:19.027+0000 7f0803836040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:08:19.036 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.034+0000 7f6f381ef640 10 monclient: get_monmap_and_config 2024-03-25T18:08:19.036 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.034+0000 7f6f381ef640 10 monclient: build_initial_monmap 2024-03-25T18:08:19.037 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.034+0000 7f6f381ef640 10 monclient: monmap: 2024-03-25T18:08:19.037 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-25T18:08:19.037 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:19.037 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:08:19.036655+0000 2024-03-25T18:08:19.037 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:08:19.036655+0000 2024-03-25T18:08:19.037 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:19.037 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:19.037 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:19.037 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:08:19.037 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:08:19.037 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:19.037 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.035+0000 7f6f381ef640 1 Processor -- start 2024-03-25T18:08:19.037 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.035+0000 7f6f381ef640 1 -- start start 2024-03-25T18:08:19.037 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.035+0000 7f6f381ef640 10 monclient: init 2024-03-25T18:08:19.038 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.035+0000 7f6f381ef640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:19.038 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.035+0000 7f6f381ef640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:08:19.038 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.035+0000 7f6f381ef640 10 monclient(hunting): picked mon.noname-a con 0x7f6f300b4700 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:19.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.035+0000 7f6f381ef640 10 monclient(hunting): picked mon.noname-b con 0x7f6f30142a20 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:19.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.035+0000 7f6f381ef640 10 monclient(hunting): picked mon.noname-c con 0x7f6f300a96b0 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:19.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.036+0000 7f6f381ef640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f300a9d60 con 0x7f6f300b4700 2024-03-25T18:08:19.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.036+0000 7f6f381ef640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f3013ff10 con 0x7f6f300a96b0 2024-03-25T18:08:19.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.036+0000 7f6f381ef640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f300aa1f0 con 0x7f6f30142a20 2024-03-25T18:08:19.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.036+0000 7f6f381ef640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:19.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.036+0000 7f6f381ef640 10 monclient(hunting): authenticate will time out at 6969.318757s 2024-03-25T18:08:19.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.036+0000 7f6f3678c640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f6f300a96b0 0x7f6f3013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:34208/0 (socket says 172.21.15.194:34208) 2024-03-25T18:08:19.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.036+0000 7f6f3678c640 1 -- 172.21.15.194:0/1953000058 learned_addr learned my addr 172.21.15.194:0/1953000058 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-25T18:08:19.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.039+0000 7f6f34f89640 1 -- 172.21.15.194:0/1953000058 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3170402850 0 0) 0x7f6f3013ff10 con 0x7f6f300a96b0 2024-03-25T18:08:19.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.039+0000 7f6f34f89640 10 monclient(hunting): _negotiate 2024-03-25T18:08:19.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.039+0000 7f6f34f89640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:19.041 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.039+0000 7f6f34f89640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:19.041 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.039+0000 7f6f34f89640 10 monclient(hunting): my global_id is 4123 2024-03-25T18:08:19.041 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.039+0000 7f6f34f89640 1 -- 172.21.15.194:0/1953000058 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f1c003520 con 0x7f6f300a96b0 2024-03-25T18:08:19.041 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.039+0000 7f6f34f89640 1 -- 172.21.15.194:0/1953000058 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 829689566 0 0) 0x7f6f300aa1f0 con 0x7f6f30142a20 2024-03-25T18:08:19.041 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.039+0000 7f6f34f89640 10 monclient(hunting): _negotiate 2024-03-25T18:08:19.041 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.039+0000 7f6f34f89640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:19.041 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.039+0000 7f6f34f89640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:19.041 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.039+0000 7f6f34f89640 10 monclient(hunting): my global_id is 4124 2024-03-25T18:08:19.041 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.039+0000 7f6f34f89640 1 -- 172.21.15.194:0/1953000058 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f3013ff10 con 0x7f6f30142a20 2024-03-25T18:08:19.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.039+0000 7f6f34f89640 1 -- 172.21.15.194:0/1953000058 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186575358 0 0) 0x7f6f300a9d60 con 0x7f6f300b4700 2024-03-25T18:08:19.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.040+0000 7f6f34f89640 10 monclient(hunting): _negotiate 2024-03-25T18:08:19.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.040+0000 7f6f34f89640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:19.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.040+0000 7f6f34f89640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:19.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.040+0000 7f6f34f89640 10 monclient(hunting): my global_id is 4122 2024-03-25T18:08:19.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.040+0000 7f6f34f89640 1 -- 172.21.15.194:0/1953000058 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f300aa1f0 con 0x7f6f300b4700 2024-03-25T18:08:19.043 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.042+0000 7f6f34f89640 1 -- 172.21.15.194:0/1953000058 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3407144528 0 0) 0x7f6f1c003520 con 0x7f6f300a96b0 2024-03-25T18:08:19.044 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.042+0000 7f6f34f89640 1 -- 172.21.15.194:0/1953000058 >> v1:172.21.15.156:6790/0 conn(0x7f6f30142a20 legacy=0x7f6f3013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:19.044 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.042+0000 7f6f34f89640 1 -- 172.21.15.194:0/1953000058 >> v1:172.21.15.156:6789/0 conn(0x7f6f300b4700 legacy=0x7f6f30142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:19.044 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.042+0000 7f6f34f89640 10 monclient: _finish_hunting 0 2024-03-25T18:08:19.044 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.042+0000 7f6f34f89640 1 monclient: found mon.noname-c 2024-03-25T18:08:19.044 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.042+0000 7f6f34f89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:19.044 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.042+0000 7f6f34f89640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:08:19.044 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.042+0000 7f6f34f89640 1 -- 172.21.15.194:0/1953000058 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f300b4ae0 con 0x7f6f300a96b0 2024-03-25T18:08:19.044 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.042+0000 7f6f34f89640 10 monclient: _finish_auth 0 2024-03-25T18:08:19.044 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.042+0000 7f6f34f89640 10 monclient: _check_auth_tickets 2024-03-25T18:08:19.044 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.042+0000 7f6f34f89640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:19.044 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.042+0000 7f6f34f89640 1 -- 172.21.15.194:0/1953000058 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f6f28001a10 con 0x7f6f300a96b0 2024-03-25T18:08:19.044 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.042+0000 7f6f34f89640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:19.044 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.042+0000 7f6f34f89640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:19.044 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.042+0000 7f6f34f89640 10 monclient: dump: 2024-03-25T18:08:19.045 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:19.045 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:19.045 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:19.045 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:19.045 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:19.045 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:19.045 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:19.045 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:19.045 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:19.046 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:19.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.042+0000 7f6f381ef640 5 monclient: authenticate success, global_id 4123 2024-03-25T18:08:19.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.042+0000 7f6f381ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:19.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.044+0000 7f6f34f89640 1 -- 172.21.15.194:0/1953000058 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f28001b70 con 0x7f6f300a96b0 2024-03-25T18:08:19.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.044+0000 7f6f34f89640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:19.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.044+0000 7f6f34f89640 1 -- 172.21.15.194:0/1953000058 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f6f28001f80 con 0x7f6f300a96b0 2024-03-25T18:08:19.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.044+0000 7f6f381ef640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:19.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.044+0000 7f6f381ef640 4 set_mon_vals no callback set 2024-03-25T18:08:19.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.044+0000 7f6f34f89640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:19.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.044+0000 7f6f34f89640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:19.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.044+0000 7f6f34f89640 10 monclient: dump: 2024-03-25T18:08:19.049 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:19.049 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:19.049 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:19.050 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:19.050 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:19.050 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:19.050 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:19.050 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:19.050 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:19.050 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:19.050 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.045+0000 7f6f381ef640 10 monclient: shutdown 2024-03-25T18:08:19.050 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.045+0000 7f6f381ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:19.050 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.045+0000 7f6f381ef640 1 -- 172.21.15.194:0/1953000058 >> v1:172.21.15.194:6789/0 conn(0x7f6f300a96b0 legacy=0x7f6f3013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:19.050 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.045+0000 7f6f381ef640 1 -- 172.21.15.194:0/1953000058 shutdown_connections 2024-03-25T18:08:19.050 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.045+0000 7f6f381ef640 1 -- 172.21.15.194:0/1953000058 >> 172.21.15.194:0/1953000058 conn(0x7f6f300668e0 msgr2=0x7f6f30066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:19.050 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.045+0000 7f6f381ef640 1 -- 172.21.15.194:0/1953000058 shutdown_connections 2024-03-25T18:08:19.050 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.045+0000 7f6f381ef640 1 -- 172.21.15.194:0/1953000058 wait complete. 2024-03-25T18:08:19.050 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.045+0000 7f6f381ef640 10 monclient: build_initial_monmap 2024-03-25T18:08:19.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.045+0000 7f6f381ef640 10 monclient: monmap: 2024-03-25T18:08:19.051 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-25T18:08:19.051 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:19.051 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 0.000000 2024-03-25T18:08:19.051 INFO:teuthology.orchestra.run.smithi194.stderr:created 0.000000 2024-03-25T18:08:19.051 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:19.051 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-25T18:08:19.051 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:19.051 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:08:19.051 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:08:19.051 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:19.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.046+0000 7f6f381ef640 1 librados: starting msgr at 2024-03-25T18:08:19.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.046+0000 7f6f381ef640 1 librados: starting objecter 2024-03-25T18:08:19.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.046+0000 7f6f381ef640 1 Processor -- start 2024-03-25T18:08:19.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.046+0000 7f6f381ef640 1 -- start start 2024-03-25T18:08:19.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.046+0000 7f6f381ef640 1 librados: setting wanted keys 2024-03-25T18:08:19.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.046+0000 7f6f381ef640 1 librados: calling monclient init 2024-03-25T18:08:19.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.046+0000 7f6f381ef640 10 monclient: init 2024-03-25T18:08:19.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.046+0000 7f6f381ef640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:19.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.046+0000 7f6f381ef640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:08:19.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.046+0000 7f6f381ef640 10 monclient(hunting): picked mon.noname-c con 0x7f6f300a96b0 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:19.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.046+0000 7f6f381ef640 10 monclient(hunting): picked mon.noname-b con 0x7f6f300b4700 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:19.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.046+0000 7f6f381ef640 10 monclient(hunting): picked mon.noname-a con 0x7f6f30142a20 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:19.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.046+0000 7f6f381ef640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f300af020 con 0x7f6f30142a20 2024-03-25T18:08:19.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.046+0000 7f6f381ef640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f300ae210 con 0x7f6f300b4700 2024-03-25T18:08:19.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.046+0000 7f6f381ef640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f300ae4f0 con 0x7f6f300a96b0 2024-03-25T18:08:19.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.046+0000 7f6f381ef640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:19.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.046+0000 7f6f381ef640 10 monclient(hunting): authenticate will time out at 6969.329607s 2024-03-25T18:08:19.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.047+0000 7f6f3578a640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f6f300b4700 0x7f6f300acc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:34216/0 (socket says 172.21.15.194:34216) 2024-03-25T18:08:19.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.047+0000 7f6f3578a640 1 -- 172.21.15.194:0/3382195901 learned_addr learned my addr 172.21.15.194:0/3382195901 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-25T18:08:19.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.047+0000 7f6f267fc640 10 client.?.objecter ms_handle_connect 0x7f6f300b4700 2024-03-25T18:08:19.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.047+0000 7f6f267fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:19.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.047+0000 7f6f267fc640 10 client.?.objecter ms_handle_connect 0x7f6f30142a20 2024-03-25T18:08:19.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.047+0000 7f6f267fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:19.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.047+0000 7f6f267fc640 10 client.?.objecter ms_handle_connect 0x7f6f300a96b0 2024-03-25T18:08:19.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.047+0000 7f6f267fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:19.053 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-25T18:08:19.047+0000 7f776ee13040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:08:19.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.049+0000 7f6f267fc640 1 -- 172.21.15.194:0/3382195901 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 305404187 0 0) 0x7f6f300ae210 con 0x7f6f300b4700 2024-03-25T18:08:19.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.049+0000 7f6f267fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:19.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.049+0000 7f6f267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:19.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.049+0000 7f6f267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:19.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.049+0000 7f6f267fc640 10 monclient(hunting): my global_id is 4126 2024-03-25T18:08:19.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.049+0000 7f6f267fc640 1 -- 172.21.15.194:0/3382195901 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f14004eb0 con 0x7f6f300b4700 2024-03-25T18:08:19.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.049+0000 7f6f267fc640 1 -- 172.21.15.194:0/3382195901 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 380286503 0 0) 0x7f6f300ae4f0 con 0x7f6f300a96b0 2024-03-25T18:08:19.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.049+0000 7f6f267fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:19.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.049+0000 7f6f267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:19.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.049+0000 7f6f267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:19.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.049+0000 7f6f267fc640 10 monclient(hunting): my global_id is 4127 2024-03-25T18:08:19.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.049+0000 7f6f267fc640 1 -- 172.21.15.194:0/3382195901 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f300ae210 con 0x7f6f300a96b0 2024-03-25T18:08:19.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.050+0000 7f6f267fc640 1 -- 172.21.15.194:0/3382195901 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3970217560 0 0) 0x7f6f300af020 con 0x7f6f30142a20 2024-03-25T18:08:19.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.050+0000 7f6f267fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:19.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.050+0000 7f6f267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:19.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.050+0000 7f6f267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:19.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.050+0000 7f6f267fc640 10 monclient(hunting): my global_id is 4125 2024-03-25T18:08:19.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.050+0000 7f6f267fc640 1 -- 172.21.15.194:0/3382195901 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f300ae4f0 con 0x7f6f30142a20 2024-03-25T18:08:19.056 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.054+0000 7f6f267fc640 1 -- 172.21.15.194:0/3382195901 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2542965097 0 0) 0x7f6f14004eb0 con 0x7f6f300b4700 2024-03-25T18:08:19.056 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.054+0000 7f6f267fc640 1 -- 172.21.15.194:0/3382195901 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f300af020 con 0x7f6f300b4700 2024-03-25T18:08:19.056 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.054+0000 7f6f267fc640 1 -- 172.21.15.194:0/3382195901 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f6f20001a30 con 0x7f6f300b4700 2024-03-25T18:08:19.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.054+0000 7f6f267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:19.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.054+0000 7f6f267fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:19.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.054+0000 7f6f267fc640 10 monclient(hunting): dump: 2024-03-25T18:08:19.057 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:19.057 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:19.057 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:19.057 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:19.057 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:19.057 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:19.057 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:19.057 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:19.057 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:19.057 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:19.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.054+0000 7f6f267fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:19.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.054+0000 7f6f267fc640 1 -- 172.21.15.194:0/3382195901 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4019468337 0 0) 0x7f6f300ae210 con 0x7f6f300a96b0 2024-03-25T18:08:19.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.054+0000 7f6f267fc640 1 -- 172.21.15.194:0/3382195901 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f14004eb0 con 0x7f6f300a96b0 2024-03-25T18:08:19.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.054+0000 7f6f267fc640 1 -- 172.21.15.194:0/3382195901 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f6f2c001930 con 0x7f6f300a96b0 2024-03-25T18:08:19.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.054+0000 7f6f267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:19.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.054+0000 7f6f267fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:19.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.054+0000 7f6f267fc640 10 monclient(hunting): dump: 2024-03-25T18:08:19.058 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:19.058 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:19.058 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:19.058 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:19.058 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:19.058 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:19.058 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:19.058 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:19.058 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:19.059 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:19.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.056+0000 7f6f267fc640 1 -- 172.21.15.194:0/3382195901 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 780323539 0 0) 0x7f6f300ae4f0 con 0x7f6f30142a20 2024-03-25T18:08:19.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.056+0000 7f6f267fc640 1 -- 172.21.15.194:0/3382195901 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f300ae210 con 0x7f6f30142a20 2024-03-25T18:08:19.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.057+0000 7f6f267fc640 1 -- 172.21.15.194:0/3382195901 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f6f28001e80 con 0x7f6f30142a20 2024-03-25T18:08:19.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.057+0000 7f6f267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:19.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.057+0000 7f6f267fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:19.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.057+0000 7f6f267fc640 10 monclient(hunting): dump: 2024-03-25T18:08:19.061 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:19.061 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:19.061 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:19.061 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:19.061 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:19.061 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:19.061 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:19.061 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:19.061 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:19.061 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:19.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.057+0000 7f6f267fc640 1 -- 172.21.15.194:0/3382195901 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3695449053 0 0) 0x7f6f300af020 con 0x7f6f300b4700 2024-03-25T18:08:19.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.057+0000 7f6f267fc640 1 -- 172.21.15.194:0/3382195901 >> v1:172.21.15.156:6790/0 conn(0x7f6f300a96b0 legacy=0x7f6f300ae910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:19.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.057+0000 7f6f267fc640 1 -- 172.21.15.194:0/3382195901 >> v1:172.21.15.156:6789/0 conn(0x7f6f30142a20 legacy=0x7f6f301e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:19.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.057+0000 7f6f267fc640 10 monclient: _finish_hunting 0 2024-03-25T18:08:19.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.057+0000 7f6f267fc640 1 monclient: found mon.b 2024-03-25T18:08:19.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.057+0000 7f6f267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:19.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.057+0000 7f6f267fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:19.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.057+0000 7f6f267fc640 1 -- 172.21.15.194:0/3382195901 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f300ac0e0 con 0x7f6f300b4700 2024-03-25T18:08:19.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.057+0000 7f6f267fc640 10 monclient: _finish_auth 0 2024-03-25T18:08:19.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.057+0000 7f6f267fc640 10 monclient: _check_auth_tickets 2024-03-25T18:08:19.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.057+0000 7f6f267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:19.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.057+0000 7f6f381ef640 5 monclient: authenticate success, global_id 4126 2024-03-25T18:08:19.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.057+0000 7f6f381ef640 10 monclient: _renew_subs 2024-03-25T18:08:19.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.057+0000 7f6f381ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:19.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.057+0000 7f6f381ef640 1 -- 172.21.15.194:0/3382195901 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f300ad340 con 0x7f6f300b4700 2024-03-25T18:08:19.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.057+0000 7f6f381ef640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:19.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.057+0000 7f6f381ef640 10 monclient: _renew_subs 2024-03-25T18:08:19.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.057+0000 7f6f381ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:19.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.057+0000 7f6f381ef640 1 -- 172.21.15.194:0/3382195901 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f3013b070 con 0x7f6f300b4700 2024-03-25T18:08:19.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.057+0000 7f6f381ef640 1 librados: init done 2024-03-25T18:08:19.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.058+0000 7f6f381ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:19.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.058+0000 7f6f381ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:19.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.058+0000 7f6f381ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:19.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.058+0000 7f6f381ef640 1 -- 172.21.15.194:0/3382195901 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f3004f730 con 0x7f6f300b4700 2024-03-25T18:08:19.104 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-25T18:08:19.104+0000 7f776ee13040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:08:19.152 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-25T18:08:19.151+0000 7f776ee13040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:08:19.251 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.249+0000 7f6f267fc640 1 -- 172.21.15.194:0/3382195901 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f20001c70 con 0x7f6f300b4700 2024-03-25T18:08:19.252 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.249+0000 7f6f267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:19.252 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.249+0000 7f6f267fc640 1 -- 172.21.15.194:0/3382195901 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f6f20003a60 con 0x7f6f300b4700 2024-03-25T18:08:19.252 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.249+0000 7f6f267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:19.252 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.249+0000 7f6f267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:19.252 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.249+0000 7f6f267fc640 10 monclient: dump: 2024-03-25T18:08:19.252 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:19.252 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:19.252 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:19.252 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:19.252 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:19.252 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:19.252 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:19.252 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:19.252 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:19.252 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:19.253 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.249+0000 7f6f267fc640 1 -- 172.21.15.194:0/3382195901 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6f20003fd0 con 0x7f6f300b4700 2024-03-25T18:08:19.253 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.249+0000 7f6f27fff640 4 set_mon_vals no callback set 2024-03-25T18:08:19.253 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.250+0000 7f6f267fc640 1 -- 172.21.15.194:0/3382195901 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 115937198 0 0) 0x7f6f20006580 con 0x7f6f300b4700 2024-03-25T18:08:19.253 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.250+0000 7f6f267fc640 10 client.4126.objecter ms_dispatch 0x7f6f30000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T18:08:19.253 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.250+0000 7f6f267fc640 3 client.4126.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T18:08:19.253 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.250+0000 7f6f267fc640 3 client.4126.objecter handle_osd_map decoding full epoch 3 2024-03-25T18:08:19.253 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.250+0000 7f6f267fc640 20 client.4126.objecter dump_active .. 0 homeless 2024-03-25T18:08:19.253 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.250+0000 7f6f267fc640 1 -- 172.21.15.194:0/3382195901 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6f200069c0 con 0x7f6f300b4700 2024-03-25T18:08:19.253 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.250+0000 7f6f267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:19.253 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.250+0000 7f6f267fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:19.379 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.377+0000 7f6f381ef640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb", "id": 1}] 2024-03-25T18:08:19.379 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.377+0000 7f6f381ef640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb", "id": 1}] 2024-03-25T18:08:19.379 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.377+0000 7f6f381ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:19.379 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.377+0000 7f6f381ef640 1 -- 172.21.15.194:0/3382195901 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd new", "uuid": "eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb", "id": 1} v 0) v1 -- 0x7f6f3013b320 con 0x7f6f300b4700 2024-03-25T18:08:19.457 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:08:19.455+0000 7f0803836040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:08:19.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.491+0000 7f6f267fc640 1 -- 172.21.15.194:0/3382195901 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2419230113 0 940483276) 0x7f6f200057c0 con 0x7f6f300b4700 2024-03-25T18:08:19.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.491+0000 7f6f267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb", "id": 1}] 2024-03-25T18:08:19.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.491+0000 7f6f267fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:19.493 INFO:teuthology.orchestra.run.smithi194.stdout:1 2024-03-25T18:08:19.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.492+0000 7f6f381ef640 10 librados: watch_flush enter 2024-03-25T18:08:19.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.492+0000 7f6f381ef640 10 librados: watch_flush exit 2024-03-25T18:08:19.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.492+0000 7f6f381ef640 20 client.4126.objecter shutdown clearing up homeless session... 2024-03-25T18:08:19.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.492+0000 7f6f381ef640 10 client.4126.objecter successfully canceled tick 2024-03-25T18:08:19.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.492+0000 7f6f381ef640 10 monclient: shutdown 2024-03-25T18:08:19.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.492+0000 7f6f381ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:19.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.492+0000 7f6f381ef640 1 -- 172.21.15.194:0/3382195901 >> v1:172.21.15.194:6789/0 conn(0x7f6f300b4700 legacy=0x7f6f300acc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:19.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.492+0000 7f6f381ef640 1 -- 172.21.15.194:0/3382195901 shutdown_connections 2024-03-25T18:08:19.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.492+0000 7f6f381ef640 1 -- 172.21.15.194:0/3382195901 >> 172.21.15.194:0/3382195901 conn(0x7f6f300668e0 msgr2=0x7f6f300b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:19.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.492+0000 7f6f381ef640 1 -- 172.21.15.194:0/3382195901 shutdown_connections 2024-03-25T18:08:19.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.492+0000 7f6f381ef640 1 -- 172.21.15.194:0/3382195901 wait complete. 2024-03-25T18:08:19.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.492+0000 7f6f381ef640 1 librados: shutdown 2024-03-25T18:08:19.505 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 069bc903-8f61-4312-a5fe-e5f02c2c695d 2 2024-03-25T18:08:19.541 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-25T18:08:19.541+0000 7f776ee13040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:08:19.599 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:08:19.598+0000 7f0803836040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:08:19.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.644+0000 7f533473d640 10 monclient: get_monmap_and_config 2024-03-25T18:08:19.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.644+0000 7f533473d640 10 monclient: build_initial_monmap 2024-03-25T18:08:19.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.644+0000 7f533473d640 10 monclient: monmap: 2024-03-25T18:08:19.646 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-25T18:08:19.646 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:19.646 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:08:19.646596+0000 2024-03-25T18:08:19.646 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:08:19.646596+0000 2024-03-25T18:08:19.646 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:19.646 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:19.646 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:19.647 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:08:19.647 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:08:19.647 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:19.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.645+0000 7f533473d640 1 Processor -- start 2024-03-25T18:08:19.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.645+0000 7f533473d640 1 -- start start 2024-03-25T18:08:19.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.645+0000 7f533473d640 10 monclient: init 2024-03-25T18:08:19.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.645+0000 7f533473d640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:19.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.645+0000 7f533473d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:08:19.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.645+0000 7f533473d640 10 monclient(hunting): picked mon.noname-a con 0x7f532c070210 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:19.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.646+0000 7f533473d640 10 monclient(hunting): picked mon.noname-c con 0x7f532c0b06d0 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:19.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.646+0000 7f533473d640 10 monclient(hunting): picked mon.noname-b con 0x7f532c0705f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:19.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.646+0000 7f533473d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f532c0b0b80 con 0x7f532c070210 2024-03-25T18:08:19.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.646+0000 7f533473d640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f532c0b0430 con 0x7f532c0b06d0 2024-03-25T18:08:19.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.646+0000 7f533473d640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f532c071670 con 0x7f532c0705f0 2024-03-25T18:08:19.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.646+0000 7f533473d640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:19.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.646+0000 7f533473d640 10 monclient(hunting): authenticate will time out at 6969.928825s 2024-03-25T18:08:19.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.646+0000 7f5331cd8640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f532c0b06d0 0x7f532c0afd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:34228/0 (socket says 172.21.15.194:34228) 2024-03-25T18:08:19.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.646+0000 7f5331cd8640 1 -- 172.21.15.194:0/4037119858 learned_addr learned my addr 172.21.15.194:0/4037119858 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-25T18:08:19.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.649+0000 7f53314d7640 1 -- 172.21.15.194:0/4037119858 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3534554139 0 0) 0x7f531c001110 con 0x7f532c0b06d0 2024-03-25T18:08:19.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.649+0000 7f53314d7640 10 monclient(hunting): _negotiate 2024-03-25T18:08:19.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.649+0000 7f53314d7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:19.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.649+0000 7f53314d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:19.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.649+0000 7f53314d7640 10 monclient(hunting): my global_id is 4129 2024-03-25T18:08:19.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.649+0000 7f53314d7640 1 -- 172.21.15.194:0/4037119858 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5318003520 con 0x7f532c0b06d0 2024-03-25T18:08:19.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.649+0000 7f53314d7640 1 -- 172.21.15.194:0/4037119858 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2714521677 0 0) 0x7f532c071670 con 0x7f532c0705f0 2024-03-25T18:08:19.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.649+0000 7f53314d7640 10 monclient(hunting): _negotiate 2024-03-25T18:08:19.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.649+0000 7f53314d7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:19.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.649+0000 7f53314d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:19.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.649+0000 7f53314d7640 10 monclient(hunting): my global_id is 4130 2024-03-25T18:08:19.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.649+0000 7f53314d7640 1 -- 172.21.15.194:0/4037119858 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f531c001110 con 0x7f532c0705f0 2024-03-25T18:08:19.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.650+0000 7f53314d7640 1 -- 172.21.15.194:0/4037119858 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058478421 0 0) 0x7f532c0b0b80 con 0x7f532c070210 2024-03-25T18:08:19.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.650+0000 7f53314d7640 10 monclient(hunting): _negotiate 2024-03-25T18:08:19.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.650+0000 7f53314d7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:19.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.650+0000 7f53314d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:19.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.650+0000 7f53314d7640 10 monclient(hunting): my global_id is 4128 2024-03-25T18:08:19.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.650+0000 7f53314d7640 1 -- 172.21.15.194:0/4037119858 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f532c071670 con 0x7f532c070210 2024-03-25T18:08:19.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.652+0000 7f53314d7640 1 -- 172.21.15.194:0/4037119858 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3537976338 0 0) 0x7f531c001110 con 0x7f532c0705f0 2024-03-25T18:08:19.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.652+0000 7f53314d7640 1 -- 172.21.15.194:0/4037119858 >> v1:172.21.15.194:6789/0 conn(0x7f532c0b06d0 legacy=0x7f532c0afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:19.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.652+0000 7f53314d7640 1 -- 172.21.15.194:0/4037119858 >> v1:172.21.15.156:6789/0 conn(0x7f532c070210 legacy=0x7f532c140930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:19.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.652+0000 7f53314d7640 10 monclient: _finish_hunting 0 2024-03-25T18:08:19.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.652+0000 7f53314d7640 1 monclient: found mon.noname-b 2024-03-25T18:08:19.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.652+0000 7f53314d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:19.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.652+0000 7f53314d7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.156:6790/0 2024-03-25T18:08:19.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.652+0000 7f53314d7640 1 -- 172.21.15.194:0/4037119858 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f532c070a60 con 0x7f532c0705f0 2024-03-25T18:08:19.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.652+0000 7f53314d7640 10 monclient: _finish_auth 0 2024-03-25T18:08:19.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.652+0000 7f53314d7640 10 monclient: _check_auth_tickets 2024-03-25T18:08:19.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.652+0000 7f53314d7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:19.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.652+0000 7f53314d7640 1 -- 172.21.15.194:0/4037119858 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5324001920 con 0x7f532c0705f0 2024-03-25T18:08:19.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.652+0000 7f533473d640 5 monclient: authenticate success, global_id 4130 2024-03-25T18:08:19.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.652+0000 7f533473d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:19.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.652+0000 7f53314d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:19.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.652+0000 7f53314d7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:19.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.652+0000 7f53314d7640 10 monclient: dump: 2024-03-25T18:08:19.655 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:19.655 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:19.655 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:19.655 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:19.655 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:19.655 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:19.655 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:19.656 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:19.656 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:19.656 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:19.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.652+0000 7f533473d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:19.658 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.654+0000 7f53314d7640 1 -- 172.21.15.194:0/4037119858 <== mon.2 v1:172.21.15.156:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5324001a80 con 0x7f532c0705f0 2024-03-25T18:08:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.654+0000 7f53314d7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.654+0000 7f53314d7640 1 -- 172.21.15.194:0/4037119858 <== mon.2 v1:172.21.15.156:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5324001e10 con 0x7f532c0705f0 2024-03-25T18:08:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.654+0000 7f53314d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.654+0000 7f53314d7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.654+0000 7f53314d7640 10 monclient: dump: 2024-03-25T18:08:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:19.660 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:19.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.654+0000 7f533473d640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:19.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.654+0000 7f533473d640 4 set_mon_vals no callback set 2024-03-25T18:08:19.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.654+0000 7f533473d640 10 monclient: shutdown 2024-03-25T18:08:19.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.654+0000 7f533473d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:19.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.654+0000 7f533473d640 1 -- 172.21.15.194:0/4037119858 >> v1:172.21.15.156:6790/0 conn(0x7f532c0705f0 legacy=0x7f532c14b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:19.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.655+0000 7f533473d640 1 -- 172.21.15.194:0/4037119858 shutdown_connections 2024-03-25T18:08:19.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.655+0000 7f533473d640 1 -- 172.21.15.194:0/4037119858 >> 172.21.15.194:0/4037119858 conn(0x7f532c0b5cd0 msgr2=0x7f532c0b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:19.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.655+0000 7f533473d640 1 -- 172.21.15.194:0/4037119858 shutdown_connections 2024-03-25T18:08:19.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.655+0000 7f533473d640 1 -- 172.21.15.194:0/4037119858 wait complete. 2024-03-25T18:08:19.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.655+0000 7f533473d640 10 monclient: build_initial_monmap 2024-03-25T18:08:19.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.655+0000 7f533473d640 10 monclient: monmap: 2024-03-25T18:08:19.660 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-25T18:08:19.660 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:19.660 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 0.000000 2024-03-25T18:08:19.660 INFO:teuthology.orchestra.run.smithi194.stderr:created 0.000000 2024-03-25T18:08:19.661 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:19.661 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-25T18:08:19.661 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:19.661 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:08:19.661 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:08:19.661 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:19.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.656+0000 7f533473d640 1 librados: starting msgr at 2024-03-25T18:08:19.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.656+0000 7f533473d640 1 librados: starting objecter 2024-03-25T18:08:19.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.656+0000 7f533473d640 1 Processor -- start 2024-03-25T18:08:19.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.656+0000 7f533473d640 1 -- start start 2024-03-25T18:08:19.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.656+0000 7f533473d640 1 librados: setting wanted keys 2024-03-25T18:08:19.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.656+0000 7f533473d640 1 librados: calling monclient init 2024-03-25T18:08:19.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.656+0000 7f533473d640 10 monclient: init 2024-03-25T18:08:19.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.656+0000 7f533473d640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:19.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.656+0000 7f533473d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:08:19.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.656+0000 7f533473d640 10 monclient(hunting): picked mon.noname-c con 0x7f532c070210 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:19.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.656+0000 7f533473d640 10 monclient(hunting): picked mon.noname-a con 0x7f532c0705f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:19.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.656+0000 7f533473d640 10 monclient(hunting): picked mon.noname-b con 0x7f532c0b06d0 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:19.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.656+0000 7f533473d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f532c1d7840 con 0x7f532c0705f0 2024-03-25T18:08:19.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.656+0000 7f533473d640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f532c06ca70 con 0x7f532c0b06d0 2024-03-25T18:08:19.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.656+0000 7f533473d640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f532c0af1c0 con 0x7f532c070210 2024-03-25T18:08:19.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.656+0000 7f533473d640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:19.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.656+0000 7f533473d640 10 monclient(hunting): authenticate will time out at 6969.939451s 2024-03-25T18:08:19.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.657+0000 7f5332cda640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f532c0b06d0 0x7f532c130ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:34236/0 (socket says 172.21.15.194:34236) 2024-03-25T18:08:19.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.657+0000 7f5332cda640 1 -- 172.21.15.194:0/2403245162 learned_addr learned my addr 172.21.15.194:0/2403245162 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-25T18:08:19.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.657+0000 7f5322ffd640 10 client.?.objecter ms_handle_connect 0x7f532c0b06d0 2024-03-25T18:08:19.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.657+0000 7f5322ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:19.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.657+0000 7f5322ffd640 10 client.?.objecter ms_handle_connect 0x7f532c070210 2024-03-25T18:08:19.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.657+0000 7f5322ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:19.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.658+0000 7f5322ffd640 10 client.?.objecter ms_handle_connect 0x7f532c0705f0 2024-03-25T18:08:19.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.658+0000 7f5322ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:19.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.659+0000 7f5322ffd640 1 -- 172.21.15.194:0/2403245162 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2949341675 0 0) 0x7f532c06ca70 con 0x7f532c0b06d0 2024-03-25T18:08:19.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.659+0000 7f5322ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:19.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.659+0000 7f5322ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:19.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.659+0000 7f5322ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:19.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.659+0000 7f5322ffd640 10 monclient(hunting): my global_id is 4132 2024-03-25T18:08:19.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.659+0000 7f5322ffd640 1 -- 172.21.15.194:0/2403245162 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5310004eb0 con 0x7f532c0b06d0 2024-03-25T18:08:19.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.659+0000 7f5322ffd640 1 -- 172.21.15.194:0/2403245162 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2889174400 0 0) 0x7f532c0af1c0 con 0x7f532c070210 2024-03-25T18:08:19.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.659+0000 7f5322ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:19.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.659+0000 7f5322ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:19.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.659+0000 7f5322ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:19.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.659+0000 7f5322ffd640 10 monclient(hunting): my global_id is 4133 2024-03-25T18:08:19.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.659+0000 7f5322ffd640 1 -- 172.21.15.194:0/2403245162 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f532c06ca70 con 0x7f532c070210 2024-03-25T18:08:19.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.660+0000 7f5322ffd640 1 -- 172.21.15.194:0/2403245162 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3347418250 0 0) 0x7f532c1d7840 con 0x7f532c0705f0 2024-03-25T18:08:19.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.660+0000 7f5322ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:19.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.660+0000 7f5322ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:19.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.660+0000 7f5322ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:19.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.660+0000 7f5322ffd640 10 monclient(hunting): my global_id is 4131 2024-03-25T18:08:19.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.660+0000 7f5322ffd640 1 -- 172.21.15.194:0/2403245162 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f532c0af1c0 con 0x7f532c0705f0 2024-03-25T18:08:19.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.664+0000 7f5322ffd640 1 -- 172.21.15.194:0/2403245162 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3799586969 0 0) 0x7f5310004eb0 con 0x7f532c0b06d0 2024-03-25T18:08:19.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.664+0000 7f5322ffd640 1 -- 172.21.15.194:0/2403245162 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f532c1d7840 con 0x7f532c0b06d0 2024-03-25T18:08:19.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.664+0000 7f5322ffd640 1 -- 172.21.15.194:0/2403245162 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5324001d90 con 0x7f532c0b06d0 2024-03-25T18:08:19.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.664+0000 7f5322ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:19.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.664+0000 7f5322ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:19.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.664+0000 7f5322ffd640 10 monclient(hunting): dump: 2024-03-25T18:08:19.666 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:19.666 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:19.666 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:19.666 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:19.666 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:19.666 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:19.666 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:19.667 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:19.667 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:19.667 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:19.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.664+0000 7f5322ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:19.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.664+0000 7f5322ffd640 1 -- 172.21.15.194:0/2403245162 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2387140715 0 0) 0x7f532c06ca70 con 0x7f532c070210 2024-03-25T18:08:19.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.664+0000 7f5322ffd640 1 -- 172.21.15.194:0/2403245162 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5310004eb0 con 0x7f532c070210 2024-03-25T18:08:19.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.664+0000 7f5322ffd640 1 -- 172.21.15.194:0/2403245162 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5328001870 con 0x7f532c070210 2024-03-25T18:08:19.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.664+0000 7f5322ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:19.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.664+0000 7f5322ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:19.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.664+0000 7f5322ffd640 10 monclient(hunting): dump: 2024-03-25T18:08:19.667 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:19.667 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:19.667 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:19.667 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:19.667 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:19.667 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:19.668 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:19.668 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:19.668 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:19.668 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:19.669 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.667+0000 7f5322ffd640 1 -- 172.21.15.194:0/2403245162 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1331371987 0 0) 0x7f532c0af1c0 con 0x7f532c0705f0 2024-03-25T18:08:19.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.667+0000 7f5322ffd640 1 -- 172.21.15.194:0/2403245162 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f532c06ca70 con 0x7f532c0705f0 2024-03-25T18:08:19.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.667+0000 7f5322ffd640 1 -- 172.21.15.194:0/2403245162 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f531c001570 con 0x7f532c0705f0 2024-03-25T18:08:19.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.667+0000 7f5322ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:19.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.667+0000 7f5322ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:19.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.667+0000 7f5322ffd640 10 monclient(hunting): dump: 2024-03-25T18:08:19.670 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:19.670 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:19.670 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:19.670 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:19.670 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:19.670 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:19.670 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:19.670 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:19.670 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:19.671 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:19.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.667+0000 7f5322ffd640 1 -- 172.21.15.194:0/2403245162 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2391852345 0 0) 0x7f532c1d7840 con 0x7f532c0b06d0 2024-03-25T18:08:19.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.667+0000 7f5322ffd640 1 -- 172.21.15.194:0/2403245162 >> v1:172.21.15.156:6790/0 conn(0x7f532c070210 legacy=0x7f532c1da8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:19.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.667+0000 7f5322ffd640 1 -- 172.21.15.194:0/2403245162 >> v1:172.21.15.156:6789/0 conn(0x7f532c0705f0 legacy=0x7f532c1d7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:19.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.667+0000 7f5322ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:08:19.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.667+0000 7f5322ffd640 1 monclient: found mon.b 2024-03-25T18:08:19.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.667+0000 7f5322ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:19.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.667+0000 7f5322ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:19.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.667+0000 7f5322ffd640 1 -- 172.21.15.194:0/2403245162 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f532c0af350 con 0x7f532c0b06d0 2024-03-25T18:08:19.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.667+0000 7f5322ffd640 10 monclient: _finish_auth 0 2024-03-25T18:08:19.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.667+0000 7f5322ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:08:19.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.667+0000 7f5322ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:19.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.667+0000 7f533473d640 5 monclient: authenticate success, global_id 4132 2024-03-25T18:08:19.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.667+0000 7f533473d640 10 monclient: _renew_subs 2024-03-25T18:08:19.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.667+0000 7f533473d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:19.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.667+0000 7f533473d640 1 -- 172.21.15.194:0/2403245162 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f532c1d7410 con 0x7f532c0b06d0 2024-03-25T18:08:19.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.667+0000 7f533473d640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:19.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.667+0000 7f533473d640 10 monclient: _renew_subs 2024-03-25T18:08:19.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.667+0000 7f533473d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:19.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.667+0000 7f533473d640 1 -- 172.21.15.194:0/2403245162 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f532c0a96d0 con 0x7f532c0b06d0 2024-03-25T18:08:19.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.667+0000 7f533473d640 1 librados: init done 2024-03-25T18:08:19.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.668+0000 7f533473d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:19.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.668+0000 7f533473d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:19.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.668+0000 7f533473d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:19.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.668+0000 7f533473d640 1 -- 172.21.15.194:0/2403245162 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f532c04f730 con 0x7f532c0b06d0 2024-03-25T18:08:19.682 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-25T18:08:19.682+0000 7f776ee13040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:08:19.712 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:08:19.710+0000 7f0803836040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:08:19.757 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:08:19.756+0000 7f0803836040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:08:19.765 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.763+0000 7f5322ffd640 1 -- 172.21.15.194:0/2403245162 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5324003c20 con 0x7f532c0b06d0 2024-03-25T18:08:19.765 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.763+0000 7f5322ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:19.765 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.763+0000 7f5322ffd640 1 -- 172.21.15.194:0/2403245162 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5324003ff0 con 0x7f532c0b06d0 2024-03-25T18:08:19.765 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.763+0000 7f5322ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:19.765 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.763+0000 7f5322ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:19.765 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.763+0000 7f5322ffd640 10 monclient: dump: 2024-03-25T18:08:19.765 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:19.765 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:19.765 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:19.765 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:19.765 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:19.765 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:19.766 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:19.766 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:19.766 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:19.766 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:19.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.763+0000 7f5322ffd640 1 -- 172.21.15.194:0/2403245162 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f53240045a0 con 0x7f532c0b06d0 2024-03-25T18:08:19.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.763+0000 7f5330cd6640 4 set_mon_vals no callback set 2024-03-25T18:08:19.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.763+0000 7f5322ffd640 1 -- 172.21.15.194:0/2403245162 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 3708982172 0 0) 0x7f5324006ac0 con 0x7f532c0b06d0 2024-03-25T18:08:19.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.763+0000 7f5322ffd640 10 client.4132.objecter ms_dispatch 0x7f532c000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T18:08:19.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.763+0000 7f5322ffd640 3 client.4132.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T18:08:19.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.763+0000 7f5322ffd640 3 client.4132.objecter handle_osd_map decoding full epoch 4 2024-03-25T18:08:19.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.763+0000 7f5322ffd640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-25T18:08:19.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.763+0000 7f5322ffd640 1 -- 172.21.15.194:0/2403245162 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5324018dc0 con 0x7f532c0b06d0 2024-03-25T18:08:19.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.763+0000 7f5322ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:19.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.763+0000 7f5322ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:19.837 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.835+0000 7f533473d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "069bc903-8f61-4312-a5fe-e5f02c2c695d", "id": 2}] 2024-03-25T18:08:19.837 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.835+0000 7f533473d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "069bc903-8f61-4312-a5fe-e5f02c2c695d", "id": 2}] 2024-03-25T18:08:19.837 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.835+0000 7f533473d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:19.837 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.835+0000 7f533473d640 1 -- 172.21.15.194:0/2403245162 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd new", "uuid": "069bc903-8f61-4312-a5fe-e5f02c2c695d", "id": 2} v 0) v1 -- 0x7f532c069f30 con 0x7f532c0b06d0 2024-03-25T18:08:19.873 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:08:19.871+0000 7f0803836040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:08:19.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.898+0000 7f5322ffd640 1 -- 172.21.15.194:0/2403245162 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "069bc903-8f61-4312-a5fe-e5f02c2c695d", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 837122970 0 216599125) 0x7f53240066a0 con 0x7f532c0b06d0 2024-03-25T18:08:19.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.898+0000 7f5322ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "069bc903-8f61-4312-a5fe-e5f02c2c695d", "id": 2}] 2024-03-25T18:08:19.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.898+0000 7f5322ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:19.899 INFO:teuthology.orchestra.run.smithi194.stdout:2 2024-03-25T18:08:19.901 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.898+0000 7f533473d640 10 librados: watch_flush enter 2024-03-25T18:08:19.901 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.898+0000 7f533473d640 10 librados: watch_flush exit 2024-03-25T18:08:19.901 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.898+0000 7f533473d640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-25T18:08:19.901 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.898+0000 7f533473d640 10 client.4132.objecter successfully canceled tick 2024-03-25T18:08:19.901 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.898+0000 7f533473d640 10 monclient: shutdown 2024-03-25T18:08:19.901 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.898+0000 7f533473d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:19.901 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.898+0000 7f533473d640 1 -- 172.21.15.194:0/2403245162 >> v1:172.21.15.194:6789/0 conn(0x7f532c0b06d0 legacy=0x7f532c130ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:19.901 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.899+0000 7f533473d640 1 -- 172.21.15.194:0/2403245162 shutdown_connections 2024-03-25T18:08:19.901 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.899+0000 7f533473d640 1 -- 172.21.15.194:0/2403245162 >> 172.21.15.194:0/2403245162 conn(0x7f532c0b5cd0 msgr2=0x7f532c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:19.901 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.899+0000 7f533473d640 1 -- 172.21.15.194:0/2403245162 shutdown_connections 2024-03-25T18:08:19.901 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.899+0000 7f533473d640 1 -- 172.21.15.194:0/2403245162 wait complete. 2024-03-25T18:08:19.901 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:19.899+0000 7f533473d640 1 librados: shutdown 2024-03-25T18:08:19.911 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 6345be70-da95-4347-b73b-80a6d24f36e4 3 2024-03-25T18:08:19.974 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:08:19.973+0000 7f0803836040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:08:20.013 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.011+0000 7fbed7f52640 10 monclient: get_monmap_and_config 2024-03-25T18:08:20.013 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.011+0000 7fbed7f52640 10 monclient: build_initial_monmap 2024-03-25T18:08:20.013 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.011+0000 7fbed7f52640 10 monclient: monmap: 2024-03-25T18:08:20.013 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-25T18:08:20.013 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:20.013 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:08:20.013619+0000 2024-03-25T18:08:20.013 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:08:20.013619+0000 2024-03-25T18:08:20.013 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:20.013 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:20.013 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:20.013 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:08:20.014 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:08:20.014 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:20.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.012+0000 7fbed7f52640 1 Processor -- start 2024-03-25T18:08:20.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.012+0000 7fbed7f52640 1 -- start start 2024-03-25T18:08:20.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.012+0000 7fbed7f52640 10 monclient: init 2024-03-25T18:08:20.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.013+0000 7fbed7f52640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:20.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.013+0000 7fbed7f52640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:08:20.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.013+0000 7fbed7f52640 10 monclient(hunting): picked mon.noname-c con 0x7fbed00b4700 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:20.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.013+0000 7fbed7f52640 10 monclient(hunting): picked mon.noname-a con 0x7fbed0142a20 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:20.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.013+0000 7fbed7f52640 10 monclient(hunting): picked mon.noname-b con 0x7fbed00a96b0 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:20.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.013+0000 7fbed7f52640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed00a9d60 con 0x7fbed0142a20 2024-03-25T18:08:20.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.013+0000 7fbed7f52640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed013ff10 con 0x7fbed00b4700 2024-03-25T18:08:20.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.013+0000 7fbed7f52640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed00aa1f0 con 0x7fbed00a96b0 2024-03-25T18:08:20.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.013+0000 7fbed7f52640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:20.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.013+0000 7fbed7f52640 10 monclient(hunting): authenticate will time out at 6970.295870s 2024-03-25T18:08:20.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.013+0000 7fbed5cee640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fbed00b4700 0x7fbed0142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:34252/0 (socket says 172.21.15.194:34252) 2024-03-25T18:08:20.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.013+0000 7fbed5cee640 1 -- 172.21.15.194:0/3646175518 learned_addr learned my addr 172.21.15.194:0/3646175518 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-25T18:08:20.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.015+0000 7fbed4cec640 1 -- 172.21.15.194:0/3646175518 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45759487 0 0) 0x7fbed013ff10 con 0x7fbed00b4700 2024-03-25T18:08:20.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.015+0000 7fbed4cec640 10 monclient(hunting): _negotiate 2024-03-25T18:08:20.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.015+0000 7fbed4cec640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:20.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.015+0000 7fbed4cec640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:20.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.015+0000 7fbed4cec640 10 monclient(hunting): my global_id is 4135 2024-03-25T18:08:20.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.015+0000 7fbed4cec640 1 -- 172.21.15.194:0/3646175518 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbebc003520 con 0x7fbed00b4700 2024-03-25T18:08:20.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.016+0000 7fbed4cec640 1 -- 172.21.15.194:0/3646175518 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222470927 0 0) 0x7fbed00aa1f0 con 0x7fbed00a96b0 2024-03-25T18:08:20.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.016+0000 7fbed4cec640 10 monclient(hunting): _negotiate 2024-03-25T18:08:20.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.016+0000 7fbed4cec640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:20.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.016+0000 7fbed4cec640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:20.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.016+0000 7fbed4cec640 10 monclient(hunting): my global_id is 4136 2024-03-25T18:08:20.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.016+0000 7fbed4cec640 1 -- 172.21.15.194:0/3646175518 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbed013ff10 con 0x7fbed00a96b0 2024-03-25T18:08:20.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.016+0000 7fbed4cec640 1 -- 172.21.15.194:0/3646175518 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3525018580 0 0) 0x7fbed00a9d60 con 0x7fbed0142a20 2024-03-25T18:08:20.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.016+0000 7fbed4cec640 10 monclient(hunting): _negotiate 2024-03-25T18:08:20.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.016+0000 7fbed4cec640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:20.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.016+0000 7fbed4cec640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:20.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.016+0000 7fbed4cec640 10 monclient(hunting): my global_id is 4134 2024-03-25T18:08:20.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.016+0000 7fbed4cec640 1 -- 172.21.15.194:0/3646175518 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbed00aa1f0 con 0x7fbed0142a20 2024-03-25T18:08:20.021 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.019+0000 7fbed4cec640 1 -- 172.21.15.194:0/3646175518 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2960590948 0 0) 0x7fbebc003520 con 0x7fbed00b4700 2024-03-25T18:08:20.021 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.019+0000 7fbed4cec640 1 -- 172.21.15.194:0/3646175518 >> v1:172.21.15.156:6790/0 conn(0x7fbed00a96b0 legacy=0x7fbed013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:20.021 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.019+0000 7fbed4cec640 1 -- 172.21.15.194:0/3646175518 >> v1:172.21.15.156:6789/0 conn(0x7fbed0142a20 legacy=0x7fbed013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:20.021 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.019+0000 7fbed4cec640 10 monclient: _finish_hunting 0 2024-03-25T18:08:20.021 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.019+0000 7fbed4cec640 1 monclient: found mon.noname-c 2024-03-25T18:08:20.021 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.019+0000 7fbed4cec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:20.021 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.019+0000 7fbed4cec640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:08:20.021 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.019+0000 7fbed4cec640 1 -- 172.21.15.194:0/3646175518 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed00b4ae0 con 0x7fbed00b4700 2024-03-25T18:08:20.021 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.019+0000 7fbed4cec640 10 monclient: _finish_auth 0 2024-03-25T18:08:20.021 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.019+0000 7fbed4cec640 10 monclient: _check_auth_tickets 2024-03-25T18:08:20.021 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.019+0000 7fbed4cec640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:20.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.019+0000 7fbed4cec640 1 -- 172.21.15.194:0/3646175518 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fbecc001690 con 0x7fbed00b4700 2024-03-25T18:08:20.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.019+0000 7fbed7f52640 5 monclient: authenticate success, global_id 4135 2024-03-25T18:08:20.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.019+0000 7fbed7f52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:20.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.019+0000 7fbed4cec640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:20.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.019+0000 7fbed4cec640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:20.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.019+0000 7fbed4cec640 10 monclient: dump: 2024-03-25T18:08:20.022 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:20.022 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:20.022 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.022 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.022 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:20.022 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:20.022 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:20.022 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:20.023 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:20.023 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:20.023 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.019+0000 7fbed7f52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:20.025 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.020+0000 7fbed4cec640 1 -- 172.21.15.194:0/3646175518 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbecc0017f0 con 0x7fbed00b4700 2024-03-25T18:08:20.026 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.020+0000 7fbed4cec640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:20.026 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.020+0000 7fbed4cec640 1 -- 172.21.15.194:0/3646175518 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fbecc001c00 con 0x7fbed00b4700 2024-03-25T18:08:20.026 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.020+0000 7fbed7f52640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:20.026 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.020+0000 7fbed7f52640 4 set_mon_vals no callback set 2024-03-25T18:08:20.026 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.020+0000 7fbed4cec640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:20.026 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.020+0000 7fbed4cec640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:20.026 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.020+0000 7fbed4cec640 10 monclient: dump: 2024-03-25T18:08:20.026 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:20.026 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:20.026 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.026 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.026 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:20.026 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:20.027 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:20.027 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:20.027 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:20.027 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:20.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.021+0000 7fbed7f52640 10 monclient: shutdown 2024-03-25T18:08:20.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.021+0000 7fbed7f52640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:20.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.021+0000 7fbed7f52640 1 -- 172.21.15.194:0/3646175518 >> v1:172.21.15.194:6789/0 conn(0x7fbed00b4700 legacy=0x7fbed0142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:20.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.021+0000 7fbed7f52640 1 -- 172.21.15.194:0/3646175518 shutdown_connections 2024-03-25T18:08:20.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.021+0000 7fbed7f52640 1 -- 172.21.15.194:0/3646175518 >> 172.21.15.194:0/3646175518 conn(0x7fbed00668e0 msgr2=0x7fbed0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:20.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.021+0000 7fbed7f52640 1 -- 172.21.15.194:0/3646175518 shutdown_connections 2024-03-25T18:08:20.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.021+0000 7fbed7f52640 1 -- 172.21.15.194:0/3646175518 wait complete. 2024-03-25T18:08:20.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.022+0000 7fbed7f52640 10 monclient: build_initial_monmap 2024-03-25T18:08:20.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.022+0000 7fbed7f52640 10 monclient: monmap: 2024-03-25T18:08:20.027 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-25T18:08:20.027 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:20.028 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 0.000000 2024-03-25T18:08:20.028 INFO:teuthology.orchestra.run.smithi194.stderr:created 0.000000 2024-03-25T18:08:20.028 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:20.028 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-25T18:08:20.028 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:20.028 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:08:20.028 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:08:20.028 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:20.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.022+0000 7fbed7f52640 1 librados: starting msgr at 2024-03-25T18:08:20.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.022+0000 7fbed7f52640 1 librados: starting objecter 2024-03-25T18:08:20.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.022+0000 7fbed7f52640 1 Processor -- start 2024-03-25T18:08:20.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.022+0000 7fbed7f52640 1 -- start start 2024-03-25T18:08:20.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.022+0000 7fbed7f52640 1 librados: setting wanted keys 2024-03-25T18:08:20.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.022+0000 7fbed7f52640 1 librados: calling monclient init 2024-03-25T18:08:20.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.022+0000 7fbed7f52640 10 monclient: init 2024-03-25T18:08:20.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.023+0000 7fbed7f52640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:20.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.023+0000 7fbed7f52640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:08:20.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.023+0000 7fbed7f52640 10 monclient(hunting): picked mon.noname-c con 0x7fbed00a96b0 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:20.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.023+0000 7fbed7f52640 10 monclient(hunting): picked mon.noname-a con 0x7fbed00b4700 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:20.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.023+0000 7fbed7f52640 10 monclient(hunting): picked mon.noname-b con 0x7fbed0142a20 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:20.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.023+0000 7fbed7f52640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed014eca0 con 0x7fbed00b4700 2024-03-25T18:08:20.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.023+0000 7fbed7f52640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed01d74d0 con 0x7fbed0142a20 2024-03-25T18:08:20.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.023+0000 7fbed7f52640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed01d8350 con 0x7fbed00a96b0 2024-03-25T18:08:20.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.023+0000 7fbed7f52640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:20.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.023+0000 7fbed7f52640 10 monclient(hunting): authenticate will time out at 6970.306173s 2024-03-25T18:08:20.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.023+0000 7fbed54ed640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fbed00b4700 0x7fbed00acb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.194:56324/0 (socket says 172.21.15.194:56324) 2024-03-25T18:08:20.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.023+0000 7fbed54ed640 1 -- 172.21.15.194:0/3846607530 learned_addr learned my addr 172.21.15.194:0/3846607530 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-25T18:08:20.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.023+0000 7fbed5cee640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7fbed00a96b0 0x7fbed01d7c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.194:35126/0 (socket says 172.21.15.194:35126) 2024-03-25T18:08:20.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.024+0000 7fbec67fc640 10 client.?.objecter ms_handle_connect 0x7fbed00b4700 2024-03-25T18:08:20.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.024+0000 7fbec67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:20.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.024+0000 7fbec67fc640 10 client.?.objecter ms_handle_connect 0x7fbed0142a20 2024-03-25T18:08:20.030 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.024+0000 7fbec67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:20.030 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.024+0000 7fbec67fc640 10 client.?.objecter ms_handle_connect 0x7fbed00a96b0 2024-03-25T18:08:20.030 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.024+0000 7fbec67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:20.031 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.026+0000 7fbec67fc640 1 -- 172.21.15.194:0/3846607530 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924744613 0 0) 0x7fbed00aa1f0 con 0x7fbed00b4700 2024-03-25T18:08:20.031 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.026+0000 7fbec67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:20.031 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.026+0000 7fbec67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:20.031 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.026+0000 7fbec67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:20.031 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.026+0000 7fbec67fc640 10 monclient(hunting): my global_id is 4137 2024-03-25T18:08:20.031 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.026+0000 7fbec67fc640 1 -- 172.21.15.194:0/3846607530 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbeb4004eb0 con 0x7fbed00b4700 2024-03-25T18:08:20.031 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.026+0000 7fbec67fc640 1 -- 172.21.15.194:0/3846607530 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1905922526 0 0) 0x7fbed01d74d0 con 0x7fbed0142a20 2024-03-25T18:08:20.031 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.026+0000 7fbec67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:20.031 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.026+0000 7fbec67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:20.031 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.026+0000 7fbec67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:20.031 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.026+0000 7fbec67fc640 10 monclient(hunting): my global_id is 4138 2024-03-25T18:08:20.032 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.026+0000 7fbec67fc640 1 -- 172.21.15.194:0/3846607530 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbed00aa1f0 con 0x7fbed0142a20 2024-03-25T18:08:20.032 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.026+0000 7fbec67fc640 1 -- 172.21.15.194:0/3846607530 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121401443 0 0) 0x7fbed01d8350 con 0x7fbed00a96b0 2024-03-25T18:08:20.032 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.026+0000 7fbec67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:20.032 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.026+0000 7fbec67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:20.032 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.026+0000 7fbec67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:20.032 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.026+0000 7fbec67fc640 10 monclient(hunting): my global_id is 4139 2024-03-25T18:08:20.032 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.026+0000 7fbec67fc640 1 -- 172.21.15.194:0/3846607530 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbed01d74d0 con 0x7fbed00a96b0 2024-03-25T18:08:20.034 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.032+0000 7fbec67fc640 1 -- 172.21.15.194:0/3846607530 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1697852926 0 0) 0x7fbed00aa1f0 con 0x7fbed0142a20 2024-03-25T18:08:20.034 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.032+0000 7fbec67fc640 1 -- 172.21.15.194:0/3846607530 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbed01d8350 con 0x7fbed0142a20 2024-03-25T18:08:20.034 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.032+0000 7fbec67fc640 1 -- 172.21.15.194:0/3846607530 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fbec8001b50 con 0x7fbed0142a20 2024-03-25T18:08:20.034 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.032+0000 7fbec67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:20.034 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.032+0000 7fbec67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:20.034 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.032+0000 7fbec67fc640 10 monclient(hunting): dump: 2024-03-25T18:08:20.034 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:20.034 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:20.034 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.034 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.035 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:20.035 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:20.035 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:20.035 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:20.035 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:20.035 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:20.035 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.032+0000 7fbec67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:20.035 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.032+0000 7fbec67fc640 1 -- 172.21.15.194:0/3846607530 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 372664529 0 0) 0x7fbeb4004eb0 con 0x7fbed00b4700 2024-03-25T18:08:20.035 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.032+0000 7fbec67fc640 1 -- 172.21.15.194:0/3846607530 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbed00aa1f0 con 0x7fbed00b4700 2024-03-25T18:08:20.035 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.032+0000 7fbec67fc640 1 -- 172.21.15.194:0/3846607530 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fbec0001870 con 0x7fbed00b4700 2024-03-25T18:08:20.035 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.032+0000 7fbec67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:20.035 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.032+0000 7fbec67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:20.035 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.032+0000 7fbec67fc640 10 monclient(hunting): dump: 2024-03-25T18:08:20.035 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:20.035 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:20.035 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.035 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.036 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:20.036 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:20.036 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:20.036 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:20.036 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:20.036 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:20.037 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.033+0000 7fbec67fc640 1 -- 172.21.15.194:0/3846607530 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2935061909 0 0) 0x7fbed01d74d0 con 0x7fbed00a96b0 2024-03-25T18:08:20.037 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.033+0000 7fbec67fc640 1 -- 172.21.15.194:0/3846607530 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbeb4004eb0 con 0x7fbed00a96b0 2024-03-25T18:08:20.038 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.033+0000 7fbec67fc640 1 -- 172.21.15.194:0/3846607530 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fbecc001b00 con 0x7fbed00a96b0 2024-03-25T18:08:20.038 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.033+0000 7fbec67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:20.038 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.033+0000 7fbec67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:20.038 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.033+0000 7fbec67fc640 10 monclient(hunting): dump: 2024-03-25T18:08:20.038 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:20.038 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:20.038 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.038 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.038 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:20.038 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:20.038 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:20.038 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:20.038 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:20.038 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:20.038 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.034+0000 7fbec67fc640 1 -- 172.21.15.194:0/3846607530 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4188078685 0 0) 0x7fbed01d8350 con 0x7fbed0142a20 2024-03-25T18:08:20.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.034+0000 7fbec67fc640 1 -- 172.21.15.194:0/3846607530 >> v1:172.21.15.156:6790/0 conn(0x7fbed00a96b0 legacy=0x7fbed01d7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:20.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.034+0000 7fbec67fc640 1 -- 172.21.15.194:0/3846607530 >> v1:172.21.15.156:6789/0 conn(0x7fbed00b4700 legacy=0x7fbed00acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:20.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.034+0000 7fbec67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:08:20.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.034+0000 7fbec67fc640 1 monclient: found mon.b 2024-03-25T18:08:20.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.034+0000 7fbec67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:20.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.034+0000 7fbec67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:20.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.034+0000 7fbec67fc640 1 -- 172.21.15.194:0/3846607530 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed00abd90 con 0x7fbed0142a20 2024-03-25T18:08:20.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.034+0000 7fbec67fc640 10 monclient: _finish_auth 0 2024-03-25T18:08:20.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.034+0000 7fbec67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:08:20.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.034+0000 7fbec67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:20.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.034+0000 7fbed7f52640 5 monclient: authenticate success, global_id 4138 2024-03-25T18:08:20.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.034+0000 7fbed7f52640 10 monclient: _renew_subs 2024-03-25T18:08:20.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.035+0000 7fbed7f52640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:20.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.035+0000 7fbed7f52640 1 -- 172.21.15.194:0/3846607530 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbed01d7720 con 0x7fbed0142a20 2024-03-25T18:08:20.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.035+0000 7fbed7f52640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:20.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.035+0000 7fbed7f52640 10 monclient: _renew_subs 2024-03-25T18:08:20.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.035+0000 7fbed7f52640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:20.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.035+0000 7fbed7f52640 1 -- 172.21.15.194:0/3846607530 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbed00ad240 con 0x7fbed0142a20 2024-03-25T18:08:20.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.035+0000 7fbed7f52640 1 librados: init done 2024-03-25T18:08:20.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.035+0000 7fbed7f52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:20.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.035+0000 7fbed7f52640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:20.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.035+0000 7fbed7f52640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:20.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.035+0000 7fbed7f52640 1 -- 172.21.15.194:0/3846607530 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbed004f730 con 0x7fbed0142a20 2024-03-25T18:08:20.054 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-25T18:08:20.053+0000 7f776ee13040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:08:20.100 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-25T18:08:20.099+0000 7f776ee13040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:08:20.138 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.136+0000 7fbec67fc640 1 -- 172.21.15.194:0/3846607530 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbec8002080 con 0x7fbed0142a20 2024-03-25T18:08:20.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.136+0000 7fbec67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:20.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.136+0000 7fbec67fc640 1 -- 172.21.15.194:0/3846607530 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fbec8003b10 con 0x7fbed0142a20 2024-03-25T18:08:20.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.136+0000 7fbec67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:20.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.136+0000 7fbec67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:20.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.136+0000 7fbec67fc640 10 monclient: dump: 2024-03-25T18:08:20.139 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:20.139 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:20.139 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.139 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.139 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:20.139 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:20.139 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:20.139 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:20.140 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:20.140 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:20.140 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.136+0000 7fbec67fc640 1 -- 172.21.15.194:0/3846607530 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fbec8004000 con 0x7fbed0142a20 2024-03-25T18:08:20.140 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.136+0000 7fbec7fff640 4 set_mon_vals no callback set 2024-03-25T18:08:20.140 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.136+0000 7fbec67fc640 1 -- 172.21.15.194:0/3846607530 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 440482746 0 0) 0x7fbec8005730 con 0x7fbed0142a20 2024-03-25T18:08:20.140 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.136+0000 7fbec67fc640 10 client.4138.objecter ms_dispatch 0x7fbed0000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T18:08:20.140 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.136+0000 7fbec67fc640 3 client.4138.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T18:08:20.140 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.136+0000 7fbec67fc640 3 client.4138.objecter handle_osd_map decoding full epoch 5 2024-03-25T18:08:20.140 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.136+0000 7fbec67fc640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-25T18:08:20.140 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.136+0000 7fbec67fc640 1 -- 172.21.15.194:0/3846607530 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fbec8005af0 con 0x7fbed0142a20 2024-03-25T18:08:20.140 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.136+0000 7fbec67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:20.140 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.136+0000 7fbec67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:20.141 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:08:20.138+0000 7f0803836040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:08:20.183 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:08:20.182+0000 7f0803836040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:08:20.212 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.210+0000 7fbed7f52640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6345be70-da95-4347-b73b-80a6d24f36e4", "id": 3}] 2024-03-25T18:08:20.212 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.210+0000 7fbed7f52640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6345be70-da95-4347-b73b-80a6d24f36e4", "id": 3}] 2024-03-25T18:08:20.212 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.210+0000 7fbed7f52640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:20.212 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.210+0000 7fbed7f52640 1 -- 172.21.15.194:0/3846607530 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd new", "uuid": "6345be70-da95-4347-b73b-80a6d24f36e4", "id": 3} v 0) v1 -- 0x7fbed00b7470 con 0x7fbed0142a20 2024-03-25T18:08:20.215 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-25T18:08:20.215+0000 7f776ee13040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:08:20.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.272+0000 7fbec67fc640 1 -- 172.21.15.194:0/3846607530 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6345be70-da95-4347-b73b-80a6d24f36e4", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 491596099 0 525046306) 0x7fbec8005220 con 0x7fbed0142a20 2024-03-25T18:08:20.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.272+0000 7fbec67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6345be70-da95-4347-b73b-80a6d24f36e4", "id": 3}] 2024-03-25T18:08:20.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.272+0000 7fbec67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:20.274 INFO:teuthology.orchestra.run.smithi194.stdout:3 2024-03-25T18:08:20.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.273+0000 7fbed7f52640 10 librados: watch_flush enter 2024-03-25T18:08:20.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.273+0000 7fbed7f52640 10 librados: watch_flush exit 2024-03-25T18:08:20.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.273+0000 7fbed7f52640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-25T18:08:20.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.273+0000 7fbed7f52640 10 client.4138.objecter successfully canceled tick 2024-03-25T18:08:20.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.273+0000 7fbed7f52640 10 monclient: shutdown 2024-03-25T18:08:20.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.273+0000 7fbed7f52640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:20.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.273+0000 7fbed7f52640 1 -- 172.21.15.194:0/3846607530 >> v1:172.21.15.194:6789/0 conn(0x7fbed0142a20 legacy=0x7fbed01d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:20.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.273+0000 7fbed7f52640 1 -- 172.21.15.194:0/3846607530 shutdown_connections 2024-03-25T18:08:20.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.273+0000 7fbed7f52640 1 -- 172.21.15.194:0/3846607530 >> 172.21.15.194:0/3846607530 conn(0x7fbed00668e0 msgr2=0x7fbed00b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:20.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.273+0000 7fbed7f52640 1 -- 172.21.15.194:0/3846607530 shutdown_connections 2024-03-25T18:08:20.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.273+0000 7fbed7f52640 1 -- 172.21.15.194:0/3846607530 wait complete. 2024-03-25T18:08:20.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.273+0000 7fbed7f52640 1 librados: shutdown 2024-03-25T18:08:20.284 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 61ad2db9-17b4-44d5-9d65-8d180c71733b 4 2024-03-25T18:08:20.329 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-25T18:08:20.329+0000 7f776ee13040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:08:20.348 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:08:20.346+0000 7f0803836040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:08:20.383 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.382+0000 7f5d87a3e640 10 monclient: get_monmap_and_config 2024-03-25T18:08:20.383 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.382+0000 7f5d87a3e640 10 monclient: build_initial_monmap 2024-03-25T18:08:20.384 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.382+0000 7f5d87a3e640 10 monclient: monmap: 2024-03-25T18:08:20.384 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-25T18:08:20.384 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:20.384 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:08:20.384215+0000 2024-03-25T18:08:20.384 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:08:20.384215+0000 2024-03-25T18:08:20.384 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:20.384 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:20.384 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:20.384 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:08:20.384 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:08:20.384 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:20.385 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.382+0000 7f5d87a3e640 1 Processor -- start 2024-03-25T18:08:20.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.383+0000 7f5d87a3e640 1 -- start start 2024-03-25T18:08:20.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.383+0000 7f5d87a3e640 10 monclient: init 2024-03-25T18:08:20.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.383+0000 7f5d87a3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:20.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.383+0000 7f5d87a3e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:08:20.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.383+0000 7f5d87a3e640 10 monclient(hunting): picked mon.noname-a con 0x7f5d800b4700 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:20.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.383+0000 7f5d87a3e640 10 monclient(hunting): picked mon.noname-c con 0x7f5d80142a20 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:20.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.383+0000 7f5d87a3e640 10 monclient(hunting): picked mon.noname-b con 0x7f5d800a96b0 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:20.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.383+0000 7f5d87a3e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d800a9d60 con 0x7f5d800b4700 2024-03-25T18:08:20.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.383+0000 7f5d87a3e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8013ff10 con 0x7f5d80142a20 2024-03-25T18:08:20.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.383+0000 7f5d87a3e640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d800aa1f0 con 0x7f5d800a96b0 2024-03-25T18:08:20.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.383+0000 7f5d87a3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:20.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.383+0000 7f5d87a3e640 10 monclient(hunting): authenticate will time out at 6970.666319s 2024-03-25T18:08:20.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.384+0000 7f5d857da640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f5d800b4700 0x7f5d80142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.194:56326/0 (socket says 172.21.15.194:56326) 2024-03-25T18:08:20.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.384+0000 7f5d857da640 1 -- 172.21.15.194:0/3421950880 learned_addr learned my addr 172.21.15.194:0/3421950880 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-25T18:08:20.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.384+0000 7f5d85fdb640 1 --1- 172.21.15.194:0/3421950880 >> v1:172.21.15.156:6790/0 conn(0x7f5d800a96b0 0x7f5d8013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.194:35140/0 (socket says 172.21.15.194:35140) 2024-03-25T18:08:20.388 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.386+0000 7f5d77fff640 1 -- 172.21.15.194:0/3421950880 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3166803249 0 0) 0x7f5d800a9d60 con 0x7f5d800b4700 2024-03-25T18:08:20.388 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.386+0000 7f5d77fff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:20.388 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.386+0000 7f5d77fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:20.388 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.386+0000 7f5d77fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:20.388 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.386+0000 7f5d77fff640 10 monclient(hunting): my global_id is 4140 2024-03-25T18:08:20.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.386+0000 7f5d77fff640 1 -- 172.21.15.194:0/3421950880 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d6c003520 con 0x7f5d800b4700 2024-03-25T18:08:20.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.386+0000 7f5d77fff640 1 -- 172.21.15.194:0/3421950880 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1163978314 0 0) 0x7f5d8013ff10 con 0x7f5d80142a20 2024-03-25T18:08:20.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.387+0000 7f5d77fff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:20.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.387+0000 7f5d77fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:20.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.387+0000 7f5d77fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:20.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.387+0000 7f5d77fff640 10 monclient(hunting): my global_id is 4141 2024-03-25T18:08:20.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.387+0000 7f5d77fff640 1 -- 172.21.15.194:0/3421950880 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d800a9d60 con 0x7f5d80142a20 2024-03-25T18:08:20.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.387+0000 7f5d77fff640 1 -- 172.21.15.194:0/3421950880 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639388819 0 0) 0x7f5d800aa1f0 con 0x7f5d800a96b0 2024-03-25T18:08:20.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.387+0000 7f5d77fff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:20.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.387+0000 7f5d77fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:20.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.387+0000 7f5d77fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:20.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.387+0000 7f5d77fff640 10 monclient(hunting): my global_id is 4142 2024-03-25T18:08:20.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.387+0000 7f5d77fff640 1 -- 172.21.15.194:0/3421950880 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d8013ff10 con 0x7f5d800a96b0 2024-03-25T18:08:20.391 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.389+0000 7f5d77fff640 1 -- 172.21.15.194:0/3421950880 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2190661018 0 0) 0x7f5d6c003520 con 0x7f5d800b4700 2024-03-25T18:08:20.391 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.389+0000 7f5d77fff640 1 -- 172.21.15.194:0/3421950880 >> v1:172.21.15.156:6790/0 conn(0x7f5d800a96b0 legacy=0x7f5d8013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:20.391 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.389+0000 7f5d77fff640 1 -- 172.21.15.194:0/3421950880 >> v1:172.21.15.194:6789/0 conn(0x7f5d80142a20 legacy=0x7f5d8013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:20.391 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.389+0000 7f5d77fff640 10 monclient: _finish_hunting 0 2024-03-25T18:08:20.391 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.389+0000 7f5d77fff640 1 monclient: found mon.noname-a 2024-03-25T18:08:20.391 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.389+0000 7f5d77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:20.391 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.389+0000 7f5d77fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:08:20.391 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.389+0000 7f5d77fff640 1 -- 172.21.15.194:0/3421950880 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d800b4ae0 con 0x7f5d800b4700 2024-03-25T18:08:20.391 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.389+0000 7f5d77fff640 10 monclient: _finish_auth 0 2024-03-25T18:08:20.391 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.389+0000 7f5d77fff640 10 monclient: _check_auth_tickets 2024-03-25T18:08:20.392 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.389+0000 7f5d77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:20.392 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.389+0000 7f5d77fff640 1 -- 172.21.15.194:0/3421950880 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5d7c001690 con 0x7f5d800b4700 2024-03-25T18:08:20.392 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.389+0000 7f5d77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:20.392 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.389+0000 7f5d77fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:08:20.392 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.389+0000 7f5d77fff640 10 monclient: dump: 2024-03-25T18:08:20.392 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:20.392 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:20.392 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.392 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.392 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:20.392 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:20.392 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:20.392 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:20.392 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:20.392 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:20.392 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.389+0000 7f5d87a3e640 5 monclient: authenticate success, global_id 4140 2024-03-25T18:08:20.392 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.389+0000 7f5d87a3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:20.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.391+0000 7f5d77fff640 1 -- 172.21.15.194:0/3421950880 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d7c0017f0 con 0x7f5d800b4700 2024-03-25T18:08:20.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.391+0000 7f5d77fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:20.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.391+0000 7f5d87a3e640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:20.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.391+0000 7f5d77fff640 1 -- 172.21.15.194:0/3421950880 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5d7c001c00 con 0x7f5d800b4700 2024-03-25T18:08:20.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.391+0000 7f5d87a3e640 4 set_mon_vals no callback set 2024-03-25T18:08:20.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.391+0000 7f5d77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:20.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.391+0000 7f5d77fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:20.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.391+0000 7f5d77fff640 10 monclient: dump: 2024-03-25T18:08:20.396 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:20.396 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:20.396 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.396 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.396 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:20.396 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:20.397 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:20.397 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:20.397 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:20.397 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:20.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.391+0000 7f5d87a3e640 10 monclient: shutdown 2024-03-25T18:08:20.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.391+0000 7f5d87a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:20.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.391+0000 7f5d87a3e640 1 -- 172.21.15.194:0/3421950880 >> v1:172.21.15.156:6789/0 conn(0x7f5d800b4700 legacy=0x7f5d80142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:20.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.392+0000 7f5d87a3e640 1 -- 172.21.15.194:0/3421950880 shutdown_connections 2024-03-25T18:08:20.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.392+0000 7f5d87a3e640 1 -- 172.21.15.194:0/3421950880 >> 172.21.15.194:0/3421950880 conn(0x7f5d800668e0 msgr2=0x7f5d80066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:20.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.392+0000 7f5d87a3e640 1 -- 172.21.15.194:0/3421950880 shutdown_connections 2024-03-25T18:08:20.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.392+0000 7f5d87a3e640 1 -- 172.21.15.194:0/3421950880 wait complete. 2024-03-25T18:08:20.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.392+0000 7f5d87a3e640 10 monclient: build_initial_monmap 2024-03-25T18:08:20.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.392+0000 7f5d87a3e640 10 monclient: monmap: 2024-03-25T18:08:20.397 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-25T18:08:20.397 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:20.397 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 0.000000 2024-03-25T18:08:20.398 INFO:teuthology.orchestra.run.smithi194.stderr:created 0.000000 2024-03-25T18:08:20.398 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:20.398 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-25T18:08:20.398 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:20.398 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:08:20.398 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:08:20.398 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:20.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.393+0000 7f5d87a3e640 1 librados: starting msgr at 2024-03-25T18:08:20.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.393+0000 7f5d87a3e640 1 librados: starting objecter 2024-03-25T18:08:20.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.393+0000 7f5d87a3e640 1 Processor -- start 2024-03-25T18:08:20.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.393+0000 7f5d87a3e640 1 -- start start 2024-03-25T18:08:20.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.393+0000 7f5d87a3e640 1 librados: setting wanted keys 2024-03-25T18:08:20.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.393+0000 7f5d87a3e640 1 librados: calling monclient init 2024-03-25T18:08:20.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.393+0000 7f5d87a3e640 10 monclient: init 2024-03-25T18:08:20.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.393+0000 7f5d87a3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:20.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.393+0000 7f5d87a3e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:08:20.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.393+0000 7f5d87a3e640 10 monclient(hunting): picked mon.noname-b con 0x7f5d800a96b0 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:20.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.393+0000 7f5d87a3e640 10 monclient(hunting): picked mon.noname-c con 0x7f5d800b4700 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:20.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.393+0000 7f5d87a3e640 10 monclient(hunting): picked mon.noname-a con 0x7f5d80142a20 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:20.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.393+0000 7f5d87a3e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d800aba20 con 0x7f5d80142a20 2024-03-25T18:08:20.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.393+0000 7f5d87a3e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d800ad830 con 0x7f5d800a96b0 2024-03-25T18:08:20.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.393+0000 7f5d87a3e640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d801db6a0 con 0x7f5d800b4700 2024-03-25T18:08:20.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.393+0000 7f5d87a3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:20.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.393+0000 7f5d87a3e640 10 monclient(hunting): authenticate will time out at 6970.676469s 2024-03-25T18:08:20.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.394+0000 7f5d85fdb640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f5d80142a20 0x7f5d801daf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.194:56332/0 (socket says 172.21.15.194:56332) 2024-03-25T18:08:20.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.394+0000 7f5d85fdb640 1 -- 172.21.15.194:0/1469620892 learned_addr learned my addr 172.21.15.194:0/1469620892 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-25T18:08:20.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.394+0000 7f5d75ffb640 10 client.?.objecter ms_handle_connect 0x7f5d80142a20 2024-03-25T18:08:20.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.394+0000 7f5d75ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:20.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.394+0000 7f5d75ffb640 10 client.?.objecter ms_handle_connect 0x7f5d800b4700 2024-03-25T18:08:20.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.394+0000 7f5d75ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:20.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.394+0000 7f5d75ffb640 10 client.?.objecter ms_handle_connect 0x7f5d800a96b0 2024-03-25T18:08:20.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.394+0000 7f5d75ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:20.400 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.396+0000 7f5d75ffb640 1 -- 172.21.15.194:0/1469620892 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373594919 0 0) 0x7f5d800aba20 con 0x7f5d80142a20 2024-03-25T18:08:20.400 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.396+0000 7f5d75ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:08:20.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.396+0000 7f5d75ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:20.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.396+0000 7f5d75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:20.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.396+0000 7f5d75ffb640 10 monclient(hunting): my global_id is 4143 2024-03-25T18:08:20.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.396+0000 7f5d75ffb640 1 -- 172.21.15.194:0/1469620892 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d64004eb0 con 0x7f5d80142a20 2024-03-25T18:08:20.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.396+0000 7f5d75ffb640 1 -- 172.21.15.194:0/1469620892 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161937900 0 0) 0x7f5d800ad830 con 0x7f5d800a96b0 2024-03-25T18:08:20.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.396+0000 7f5d75ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:08:20.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.396+0000 7f5d75ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:20.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.396+0000 7f5d75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:20.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.396+0000 7f5d75ffb640 10 monclient(hunting): my global_id is 4144 2024-03-25T18:08:20.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.396+0000 7f5d75ffb640 1 -- 172.21.15.194:0/1469620892 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d800aba20 con 0x7f5d800a96b0 2024-03-25T18:08:20.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.397+0000 7f5d75ffb640 1 -- 172.21.15.194:0/1469620892 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 827583592 0 0) 0x7f5d801db6a0 con 0x7f5d800b4700 2024-03-25T18:08:20.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.397+0000 7f5d75ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:08:20.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.397+0000 7f5d75ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:20.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.397+0000 7f5d75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:20.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.397+0000 7f5d75ffb640 10 monclient(hunting): my global_id is 4145 2024-03-25T18:08:20.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.397+0000 7f5d75ffb640 1 -- 172.21.15.194:0/1469620892 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d800ad830 con 0x7f5d800b4700 2024-03-25T18:08:20.402 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.401+0000 7f5d75ffb640 1 -- 172.21.15.194:0/1469620892 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 290838987 0 0) 0x7f5d64004eb0 con 0x7f5d80142a20 2024-03-25T18:08:20.402 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.401+0000 7f5d75ffb640 1 -- 172.21.15.194:0/1469620892 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d801db6a0 con 0x7f5d80142a20 2024-03-25T18:08:20.402 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.401+0000 7f5d75ffb640 1 -- 172.21.15.194:0/1469620892 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5d78001c10 con 0x7f5d80142a20 2024-03-25T18:08:20.403 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.401+0000 7f5d75ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:20.403 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.401+0000 7f5d75ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:08:20.403 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.401+0000 7f5d75ffb640 10 monclient(hunting): dump: 2024-03-25T18:08:20.403 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:20.403 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:20.403 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.403 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.403 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:20.403 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:20.403 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:20.403 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:20.403 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:20.403 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:20.403 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.401+0000 7f5d75ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:20.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.401+0000 7f5d75ffb640 1 -- 172.21.15.194:0/1469620892 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 414574913 0 0) 0x7f5d800aba20 con 0x7f5d800a96b0 2024-03-25T18:08:20.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.401+0000 7f5d75ffb640 1 -- 172.21.15.194:0/1469620892 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d64004eb0 con 0x7f5d800a96b0 2024-03-25T18:08:20.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.401+0000 7f5d75ffb640 1 -- 172.21.15.194:0/1469620892 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5d7c001b00 con 0x7f5d800a96b0 2024-03-25T18:08:20.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.401+0000 7f5d75ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:20.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.401+0000 7f5d75ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:20.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.401+0000 7f5d75ffb640 10 monclient(hunting): dump: 2024-03-25T18:08:20.404 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:20.404 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:20.404 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.404 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.404 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:20.404 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:20.404 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:20.405 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:20.405 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:20.405 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:20.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.404+0000 7f5d75ffb640 1 -- 172.21.15.194:0/1469620892 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2840193630 0 0) 0x7f5d800ad830 con 0x7f5d800b4700 2024-03-25T18:08:20.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.404+0000 7f5d75ffb640 1 -- 172.21.15.194:0/1469620892 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d800aba20 con 0x7f5d800b4700 2024-03-25T18:08:20.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.404+0000 7f5d75ffb640 1 -- 172.21.15.194:0/1469620892 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5d700018f0 con 0x7f5d800b4700 2024-03-25T18:08:20.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.404+0000 7f5d75ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:20.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.404+0000 7f5d75ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:20.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.404+0000 7f5d75ffb640 10 monclient(hunting): dump: 2024-03-25T18:08:20.406 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:20.406 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:20.406 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.406 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.406 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:20.407 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:20.407 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:20.407 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:20.407 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:20.407 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:20.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.404+0000 7f5d75ffb640 1 -- 172.21.15.194:0/1469620892 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1621631194 0 0) 0x7f5d801db6a0 con 0x7f5d80142a20 2024-03-25T18:08:20.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.404+0000 7f5d75ffb640 1 -- 172.21.15.194:0/1469620892 >> v1:172.21.15.156:6790/0 conn(0x7f5d800b4700 legacy=0x7f5d801da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:20.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.404+0000 7f5d75ffb640 1 -- 172.21.15.194:0/1469620892 >> v1:172.21.15.194:6789/0 conn(0x7f5d800a96b0 legacy=0x7f5d801d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:20.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.404+0000 7f5d75ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:08:20.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.404+0000 7f5d75ffb640 1 monclient: found mon.a 2024-03-25T18:08:20.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.404+0000 7f5d75ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:20.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.404+0000 7f5d75ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:08:20.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.404+0000 7f5d75ffb640 1 -- 172.21.15.194:0/1469620892 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d801d4e90 con 0x7f5d80142a20 2024-03-25T18:08:20.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.404+0000 7f5d75ffb640 10 monclient: _finish_auth 0 2024-03-25T18:08:20.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.404+0000 7f5d75ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:08:20.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.404+0000 7f5d75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:20.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.404+0000 7f5d87a3e640 5 monclient: authenticate success, global_id 4143 2024-03-25T18:08:20.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.404+0000 7f5d87a3e640 10 monclient: _renew_subs 2024-03-25T18:08:20.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.404+0000 7f5d87a3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:08:20.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.404+0000 7f5d87a3e640 1 -- 172.21.15.194:0/1469620892 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d8013ba80 con 0x7f5d80142a20 2024-03-25T18:08:20.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.404+0000 7f5d87a3e640 10 client.4143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:20.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.404+0000 7f5d87a3e640 10 monclient: _renew_subs 2024-03-25T18:08:20.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.404+0000 7f5d87a3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:08:20.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.404+0000 7f5d87a3e640 1 -- 172.21.15.194:0/1469620892 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d800ac790 con 0x7f5d80142a20 2024-03-25T18:08:20.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.404+0000 7f5d87a3e640 1 librados: init done 2024-03-25T18:08:20.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.405+0000 7f5d87a3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:20.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.405+0000 7f5d87a3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:20.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.405+0000 7f5d87a3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:08:20.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.405+0000 7f5d87a3e640 1 -- 172.21.15.194:0/1469620892 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d800ab880 con 0x7f5d80142a20 2024-03-25T18:08:20.495 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-25T18:08:20.495+0000 7f776ee13040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:08:20.503 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.503+0000 7f5d75ffb640 1 -- 172.21.15.194:0/1469620892 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d78004b60 con 0x7f5d80142a20 2024-03-25T18:08:20.503 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.503+0000 7f5d75ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:20.503 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.503+0000 7f5d75ffb640 1 -- 172.21.15.194:0/1469620892 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5d78004cd0 con 0x7f5d80142a20 2024-03-25T18:08:20.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.503+0000 7f5d75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:20.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.503+0000 7f5d75ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:20.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.503+0000 7f5d75ffb640 10 monclient: dump: 2024-03-25T18:08:20.504 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:20.504 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:20.504 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.504 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.504 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:20.504 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:20.504 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:20.504 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:20.504 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:20.504 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:20.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.503+0000 7f5d75ffb640 1 -- 172.21.15.194:0/1469620892 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5d780051c0 con 0x7f5d80142a20 2024-03-25T18:08:20.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.503+0000 7f5d777fe640 4 set_mon_vals no callback set 2024-03-25T18:08:20.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.504+0000 7f5d75ffb640 1 -- 172.21.15.194:0/1469620892 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2902896351 0 0) 0x7f5d78006790 con 0x7f5d80142a20 2024-03-25T18:08:20.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.504+0000 7f5d75ffb640 10 client.4143.objecter ms_dispatch 0x7f5d80000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T18:08:20.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.504+0000 7f5d75ffb640 3 client.4143.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T18:08:20.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.504+0000 7f5d75ffb640 3 client.4143.objecter handle_osd_map decoding full epoch 6 2024-03-25T18:08:20.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.504+0000 7f5d75ffb640 20 client.4143.objecter dump_active .. 0 homeless 2024-03-25T18:08:20.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.504+0000 7f5d75ffb640 1 -- 172.21.15.194:0/1469620892 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5d78006b00 con 0x7f5d80142a20 2024-03-25T18:08:20.506 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.504+0000 7f5d75ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:20.506 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.504+0000 7f5d75ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:20.538 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-25T18:08:20.537+0000 7f776ee13040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:08:20.575 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.574+0000 7f5d87a3e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "61ad2db9-17b4-44d5-9d65-8d180c71733b", "id": 4}] 2024-03-25T18:08:20.575 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.574+0000 7f5d87a3e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "61ad2db9-17b4-44d5-9d65-8d180c71733b", "id": 4}] 2024-03-25T18:08:20.575 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.574+0000 7f5d87a3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:08:20.575 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.574+0000 7f5d87a3e640 1 -- 172.21.15.194:0/1469620892 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd new", "uuid": "61ad2db9-17b4-44d5-9d65-8d180c71733b", "id": 4} v 0) v1 -- 0x7f5d8004f730 con 0x7f5d80142a20 2024-03-25T18:08:20.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.620+0000 7f5d75ffb640 1 -- 172.21.15.194:0/1469620892 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "61ad2db9-17b4-44d5-9d65-8d180c71733b", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 4115521950 0 1697012583) 0x7f5d78006310 con 0x7f5d80142a20 2024-03-25T18:08:20.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.620+0000 7f5d75ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "61ad2db9-17b4-44d5-9d65-8d180c71733b", "id": 4}] 2024-03-25T18:08:20.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.620+0000 7f5d75ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:20.621 INFO:teuthology.orchestra.run.smithi194.stdout:4 2024-03-25T18:08:20.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.621+0000 7f5d87a3e640 10 librados: watch_flush enter 2024-03-25T18:08:20.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.621+0000 7f5d87a3e640 10 librados: watch_flush exit 2024-03-25T18:08:20.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.621+0000 7f5d87a3e640 20 client.4143.objecter shutdown clearing up homeless session... 2024-03-25T18:08:20.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.621+0000 7f5d87a3e640 10 client.4143.objecter successfully canceled tick 2024-03-25T18:08:20.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.621+0000 7f5d87a3e640 10 monclient: shutdown 2024-03-25T18:08:20.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.621+0000 7f5d87a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:20.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.621+0000 7f5d87a3e640 1 -- 172.21.15.194:0/1469620892 >> v1:172.21.15.156:6789/0 conn(0x7f5d80142a20 legacy=0x7f5d801daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:20.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.621+0000 7f5d87a3e640 1 -- 172.21.15.194:0/1469620892 shutdown_connections 2024-03-25T18:08:20.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.621+0000 7f5d87a3e640 1 -- 172.21.15.194:0/1469620892 >> 172.21.15.194:0/1469620892 conn(0x7f5d800668e0 msgr2=0x7f5d800b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:20.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.621+0000 7f5d87a3e640 1 -- 172.21.15.194:0/1469620892 shutdown_connections 2024-03-25T18:08:20.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.621+0000 7f5d87a3e640 1 -- 172.21.15.194:0/1469620892 wait complete. 2024-03-25T18:08:20.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.621+0000 7f5d87a3e640 1 librados: shutdown 2024-03-25T18:08:20.632 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1 5 2024-03-25T18:08:20.709 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-25T18:08:20.708+0000 7f776ee13040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:08:20.732 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.731+0000 7f0ce5cd9640 10 monclient: get_monmap_and_config 2024-03-25T18:08:20.732 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.731+0000 7f0ce5cd9640 10 monclient: build_initial_monmap 2024-03-25T18:08:20.732 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.731+0000 7f0ce5cd9640 10 monclient: monmap: 2024-03-25T18:08:20.732 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-25T18:08:20.732 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:20.732 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:08:20.732861+0000 2024-03-25T18:08:20.733 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:08:20.732861+0000 2024-03-25T18:08:20.733 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:20.733 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:20.733 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:20.733 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:08:20.733 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:08:20.733 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:20.734 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.732+0000 7f0ce5cd9640 1 Processor -- start 2024-03-25T18:08:20.735 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.732+0000 7f0ce5cd9640 1 -- start start 2024-03-25T18:08:20.735 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.732+0000 7f0ce5cd9640 10 monclient: init 2024-03-25T18:08:20.735 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.733+0000 7f0ce5cd9640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:20.735 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.733+0000 7f0ce5cd9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:08:20.735 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.733+0000 7f0ce5cd9640 10 monclient(hunting): picked mon.noname-c con 0x7f0ce00b4700 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:20.735 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.733+0000 7f0ce5cd9640 10 monclient(hunting): picked mon.noname-b con 0x7f0ce0142a20 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:20.735 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.733+0000 7f0ce5cd9640 10 monclient(hunting): picked mon.noname-a con 0x7f0ce00a96b0 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:20.735 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.733+0000 7f0ce5cd9640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce00a9d60 con 0x7f0ce00a96b0 2024-03-25T18:08:20.735 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.733+0000 7f0ce5cd9640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce013ff10 con 0x7f0ce00b4700 2024-03-25T18:08:20.735 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.733+0000 7f0ce5cd9640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce00aa1f0 con 0x7f0ce0142a20 2024-03-25T18:08:20.735 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.733+0000 7f0ce5cd9640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:20.735 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.733+0000 7f0ce5cd9640 10 monclient(hunting): authenticate will time out at 6971.015145s 2024-03-25T18:08:20.735 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.733+0000 7f0cdffff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f0ce00a96b0 0x7f0ce013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.194:56340/0 (socket says 172.21.15.194:56340) 2024-03-25T18:08:20.735 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.733+0000 7f0cdffff640 1 -- 172.21.15.194:0/3227568395 learned_addr learned my addr 172.21.15.194:0/3227568395 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-25T18:08:20.736 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.736+0000 7f0cde7fc640 1 -- 172.21.15.194:0/3227568395 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3570097352 0 0) 0x7f0ce00a9d60 con 0x7f0ce00a96b0 2024-03-25T18:08:20.736 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.736+0000 7f0cde7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:20.736 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.736+0000 7f0cde7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:20.737 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.736+0000 7f0cde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:20.737 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.736+0000 7f0cde7fc640 10 monclient(hunting): my global_id is 4146 2024-03-25T18:08:20.737 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.736+0000 7f0cde7fc640 1 -- 172.21.15.194:0/3227568395 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cc8003520 con 0x7f0ce00a96b0 2024-03-25T18:08:20.737 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.736+0000 7f0cde7fc640 1 -- 172.21.15.194:0/3227568395 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3117319721 0 0) 0x7f0ce013ff10 con 0x7f0ce00b4700 2024-03-25T18:08:20.738 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.736+0000 7f0cde7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:20.738 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.736+0000 7f0cde7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:20.738 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.736+0000 7f0cde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:20.738 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.736+0000 7f0cde7fc640 10 monclient(hunting): my global_id is 4147 2024-03-25T18:08:20.738 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.736+0000 7f0cde7fc640 1 -- 172.21.15.194:0/3227568395 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce00a9d60 con 0x7f0ce00b4700 2024-03-25T18:08:20.738 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.737+0000 7f0cde7fc640 1 -- 172.21.15.194:0/3227568395 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3929292219 0 0) 0x7f0ce00aa1f0 con 0x7f0ce0142a20 2024-03-25T18:08:20.738 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.737+0000 7f0cde7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:20.738 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.737+0000 7f0cde7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:20.738 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.737+0000 7f0cde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:20.738 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.737+0000 7f0cde7fc640 10 monclient(hunting): my global_id is 4148 2024-03-25T18:08:20.738 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.737+0000 7f0cde7fc640 1 -- 172.21.15.194:0/3227568395 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce013ff10 con 0x7f0ce0142a20 2024-03-25T18:08:20.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.739+0000 7f0cde7fc640 1 -- 172.21.15.194:0/3227568395 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2936085397 0 0) 0x7f0cc8003520 con 0x7f0ce00a96b0 2024-03-25T18:08:20.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.739+0000 7f0cde7fc640 1 -- 172.21.15.194:0/3227568395 >> v1:172.21.15.156:6790/0 conn(0x7f0ce0142a20 legacy=0x7f0ce013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:20.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.739+0000 7f0cde7fc640 1 -- 172.21.15.194:0/3227568395 >> v1:172.21.15.194:6789/0 conn(0x7f0ce00b4700 legacy=0x7f0ce0142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:20.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.739+0000 7f0cde7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:08:20.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.739+0000 7f0cde7fc640 1 monclient: found mon.noname-a 2024-03-25T18:08:20.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.739+0000 7f0cde7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:20.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.739+0000 7f0cde7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:08:20.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.739+0000 7f0cde7fc640 1 -- 172.21.15.194:0/3227568395 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce00b4ae0 con 0x7f0ce00a96b0 2024-03-25T18:08:20.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.739+0000 7f0cde7fc640 10 monclient: _finish_auth 0 2024-03-25T18:08:20.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.739+0000 7f0cde7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:08:20.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.739+0000 7f0cde7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:20.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.739+0000 7f0cde7fc640 1 -- 172.21.15.194:0/3227568395 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f0cd8001a10 con 0x7f0ce00a96b0 2024-03-25T18:08:20.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.739+0000 7f0cde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:20.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.739+0000 7f0cde7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:08:20.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.739+0000 7f0cde7fc640 10 monclient: dump: 2024-03-25T18:08:20.741 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:20.741 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:20.741 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.741 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.741 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:20.741 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:20.741 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:20.741 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:20.741 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:20.741 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:20.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.739+0000 7f0ce5cd9640 5 monclient: authenticate success, global_id 4146 2024-03-25T18:08:20.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.739+0000 7f0ce5cd9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:20.743 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.741+0000 7f0cde7fc640 1 -- 172.21.15.194:0/3227568395 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cd8001b70 con 0x7f0ce00a96b0 2024-03-25T18:08:20.743 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.741+0000 7f0cde7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:20.743 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.741+0000 7f0cde7fc640 1 -- 172.21.15.194:0/3227568395 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f0cd8001f80 con 0x7f0ce00a96b0 2024-03-25T18:08:20.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.741+0000 7f0ce5cd9640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:20.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.741+0000 7f0ce5cd9640 4 set_mon_vals no callback set 2024-03-25T18:08:20.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.741+0000 7f0cde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:20.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.741+0000 7f0cde7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:20.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.741+0000 7f0cde7fc640 10 monclient: dump: 2024-03-25T18:08:20.744 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:20.744 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:20.744 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.744 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.744 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:20.744 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:20.744 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:20.744 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:20.745 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:20.745 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:20.745 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.741+0000 7f0ce5cd9640 10 monclient: shutdown 2024-03-25T18:08:20.745 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.741+0000 7f0ce5cd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:20.745 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.741+0000 7f0ce5cd9640 1 -- 172.21.15.194:0/3227568395 >> v1:172.21.15.156:6789/0 conn(0x7f0ce00a96b0 legacy=0x7f0ce013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:20.745 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.742+0000 7f0ce5cd9640 1 -- 172.21.15.194:0/3227568395 shutdown_connections 2024-03-25T18:08:20.745 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.742+0000 7f0ce5cd9640 1 -- 172.21.15.194:0/3227568395 >> 172.21.15.194:0/3227568395 conn(0x7f0ce00668e0 msgr2=0x7f0ce0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:20.745 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.742+0000 7f0ce5cd9640 1 -- 172.21.15.194:0/3227568395 shutdown_connections 2024-03-25T18:08:20.745 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.742+0000 7f0ce5cd9640 1 -- 172.21.15.194:0/3227568395 wait complete. 2024-03-25T18:08:20.745 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.742+0000 7f0ce5cd9640 10 monclient: build_initial_monmap 2024-03-25T18:08:20.745 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.742+0000 7f0ce5cd9640 10 monclient: monmap: 2024-03-25T18:08:20.745 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-25T18:08:20.745 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:20.745 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 0.000000 2024-03-25T18:08:20.745 INFO:teuthology.orchestra.run.smithi194.stderr:created 0.000000 2024-03-25T18:08:20.745 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:20.746 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-25T18:08:20.746 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:20.746 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:08:20.746 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:08:20.746 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:20.746 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.742+0000 7f0ce5cd9640 1 librados: starting msgr at 2024-03-25T18:08:20.746 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.742+0000 7f0ce5cd9640 1 librados: starting objecter 2024-03-25T18:08:20.746 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.743+0000 7f0ce5cd9640 1 Processor -- start 2024-03-25T18:08:20.746 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.743+0000 7f0ce5cd9640 1 -- start start 2024-03-25T18:08:20.746 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.743+0000 7f0ce5cd9640 1 librados: setting wanted keys 2024-03-25T18:08:20.746 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.743+0000 7f0ce5cd9640 1 librados: calling monclient init 2024-03-25T18:08:20.746 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.743+0000 7f0ce5cd9640 10 monclient: init 2024-03-25T18:08:20.749 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.743+0000 7f0ce5cd9640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:20.749 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.743+0000 7f0ce5cd9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:08:20.749 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.743+0000 7f0ce5cd9640 10 monclient(hunting): picked mon.noname-b con 0x7f0ce00a96b0 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:20.749 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.743+0000 7f0ce5cd9640 10 monclient(hunting): picked mon.noname-c con 0x7f0ce00b4700 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:20.749 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.743+0000 7f0ce5cd9640 10 monclient(hunting): picked mon.noname-a con 0x7f0ce0142a20 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:20.749 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.743+0000 7f0ce5cd9640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce00b1640 con 0x7f0ce0142a20 2024-03-25T18:08:20.749 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.743+0000 7f0ce5cd9640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce00ae110 con 0x7f0ce00a96b0 2024-03-25T18:08:20.749 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.743+0000 7f0ce5cd9640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce0139b80 con 0x7f0ce00b4700 2024-03-25T18:08:20.749 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.743+0000 7f0ce5cd9640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:20.749 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.743+0000 7f0ce5cd9640 10 monclient(hunting): authenticate will time out at 6971.025533s 2024-03-25T18:08:20.749 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.744+0000 7f0cdffff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f0ce0142a20 0x7f0ce013a7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.194:56352/0 (socket says 172.21.15.194:56352) 2024-03-25T18:08:20.750 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.744+0000 7f0cdeffd640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f0ce00b4700 0x7f0ce013a0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.194:35164/0 (socket says 172.21.15.194:35164) 2024-03-25T18:08:20.750 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.744+0000 7f0cdffff640 1 -- 172.21.15.194:0/3696343388 learned_addr learned my addr 172.21.15.194:0/3696343388 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-25T18:08:20.750 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.744+0000 7f0cc7fff640 10 client.?.objecter ms_handle_connect 0x7f0ce0142a20 2024-03-25T18:08:20.750 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.744+0000 7f0cc7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:20.750 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.744+0000 7f0cc7fff640 10 client.?.objecter ms_handle_connect 0x7f0ce00b4700 2024-03-25T18:08:20.750 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.744+0000 7f0cc7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:20.750 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.744+0000 7f0cc7fff640 10 client.?.objecter ms_handle_connect 0x7f0ce00a96b0 2024-03-25T18:08:20.750 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.744+0000 7f0cc7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:20.750 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.746+0000 7f0cc7fff640 1 -- 172.21.15.194:0/3696343388 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498267835 0 0) 0x7f0ce00b4ae0 con 0x7f0ce0142a20 2024-03-25T18:08:20.750 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.746+0000 7f0cc7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:20.750 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.746+0000 7f0cc7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:20.750 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.746+0000 7f0cc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:20.750 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.746+0000 7f0cc7fff640 10 monclient(hunting): my global_id is 4149 2024-03-25T18:08:20.750 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.746+0000 7f0cc7fff640 1 -- 172.21.15.194:0/3696343388 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ccc004eb0 con 0x7f0ce0142a20 2024-03-25T18:08:20.750 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.746+0000 7f0cc7fff640 1 -- 172.21.15.194:0/3696343388 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2876312784 0 0) 0x7f0ce00ae110 con 0x7f0ce00a96b0 2024-03-25T18:08:20.750 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.746+0000 7f0cc7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:20.751 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.746+0000 7f0cc7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:20.751 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.746+0000 7f0cc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:20.751 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.746+0000 7f0cc7fff640 10 monclient(hunting): my global_id is 4150 2024-03-25T18:08:20.751 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.746+0000 7f0cc7fff640 1 -- 172.21.15.194:0/3696343388 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce00b4ae0 con 0x7f0ce00a96b0 2024-03-25T18:08:20.751 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.747+0000 7f0cc7fff640 1 -- 172.21.15.194:0/3696343388 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 118672034 0 0) 0x7f0ce0139b80 con 0x7f0ce00b4700 2024-03-25T18:08:20.751 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.747+0000 7f0cc7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:20.751 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.747+0000 7f0cc7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:20.751 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.747+0000 7f0cc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:20.751 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.747+0000 7f0cc7fff640 10 monclient(hunting): my global_id is 4151 2024-03-25T18:08:20.751 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.747+0000 7f0cc7fff640 1 -- 172.21.15.194:0/3696343388 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce00ae110 con 0x7f0ce00b4700 2024-03-25T18:08:20.752 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.751+0000 7f0cc7fff640 1 -- 172.21.15.194:0/3696343388 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 306329773 0 0) 0x7f0ccc004eb0 con 0x7f0ce0142a20 2024-03-25T18:08:20.752 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.751+0000 7f0cc7fff640 1 -- 172.21.15.194:0/3696343388 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ce0139b80 con 0x7f0ce0142a20 2024-03-25T18:08:20.752 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.751+0000 7f0cc7fff640 1 -- 172.21.15.194:0/3696343388 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f0cd8001dc0 con 0x7f0ce0142a20 2024-03-25T18:08:20.752 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.751+0000 7f0cc7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:20.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.751+0000 7f0cc7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:08:20.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.751+0000 7f0cc7fff640 10 monclient(hunting): dump: 2024-03-25T18:08:20.753 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:20.753 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:20.753 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.753 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.753 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:20.753 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:20.753 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:20.753 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:20.753 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:20.753 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:20.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.751+0000 7f0cc7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:20.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.752+0000 7f0cc7fff640 1 -- 172.21.15.194:0/3696343388 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1487575927 0 0) 0x7f0ce00b4ae0 con 0x7f0ce00a96b0 2024-03-25T18:08:20.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.752+0000 7f0cc7fff640 1 -- 172.21.15.194:0/3696343388 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ccc004eb0 con 0x7f0ce00a96b0 2024-03-25T18:08:20.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.752+0000 7f0cc7fff640 1 -- 172.21.15.194:0/3696343388 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f0cd4001970 con 0x7f0ce00a96b0 2024-03-25T18:08:20.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.752+0000 7f0cc7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:20.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.752+0000 7f0cc7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:20.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.752+0000 7f0cc7fff640 10 monclient(hunting): dump: 2024-03-25T18:08:20.754 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:20.754 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:20.754 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.754 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.754 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:20.754 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:20.754 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:20.755 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:20.755 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:20.755 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:20.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.754+0000 7f0cc7fff640 1 -- 172.21.15.194:0/3696343388 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3052001839 0 0) 0x7f0ce00ae110 con 0x7f0ce00b4700 2024-03-25T18:08:20.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.754+0000 7f0cc7fff640 1 -- 172.21.15.194:0/3696343388 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ce00b4ae0 con 0x7f0ce00b4700 2024-03-25T18:08:20.757 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.754+0000 7f0cc7fff640 1 -- 172.21.15.194:0/3696343388 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f0cd0001850 con 0x7f0ce00b4700 2024-03-25T18:08:20.757 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.754+0000 7f0cc7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:20.757 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.754+0000 7f0cc7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:20.757 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.754+0000 7f0cc7fff640 10 monclient(hunting): dump: 2024-03-25T18:08:20.757 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:20.757 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:20.757 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.757 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.757 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:20.757 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:20.757 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:20.757 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:20.757 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:20.757 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:20.757 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.754+0000 7f0cc7fff640 1 -- 172.21.15.194:0/3696343388 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3297795950 0 0) 0x7f0ce0139b80 con 0x7f0ce0142a20 2024-03-25T18:08:20.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.754+0000 7f0cc7fff640 1 -- 172.21.15.194:0/3696343388 >> v1:172.21.15.156:6790/0 conn(0x7f0ce00b4700 legacy=0x7f0ce013a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:20.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.754+0000 7f0cc7fff640 1 -- 172.21.15.194:0/3696343388 >> v1:172.21.15.194:6789/0 conn(0x7f0ce00a96b0 legacy=0x7f0ce0135ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:20.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.754+0000 7f0cc7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:08:20.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.754+0000 7f0cc7fff640 1 monclient: found mon.a 2024-03-25T18:08:20.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.754+0000 7f0cc7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:20.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.754+0000 7f0cc7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:08:20.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.754+0000 7f0cc7fff640 1 -- 172.21.15.194:0/3696343388 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce01362b0 con 0x7f0ce0142a20 2024-03-25T18:08:20.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.754+0000 7f0cc7fff640 10 monclient: _finish_auth 0 2024-03-25T18:08:20.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.754+0000 7f0cc7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:08:20.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.754+0000 7f0cc7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:20.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.754+0000 7f0ce5cd9640 5 monclient: authenticate success, global_id 4149 2024-03-25T18:08:20.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.754+0000 7f0ce5cd9640 10 monclient: _renew_subs 2024-03-25T18:08:20.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.754+0000 7f0ce5cd9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:08:20.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.754+0000 7f0ce5cd9640 1 -- 172.21.15.194:0/3696343388 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce01e34f0 con 0x7f0ce0142a20 2024-03-25T18:08:20.759 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.754+0000 7f0ce5cd9640 10 client.4149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:20.759 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.754+0000 7f0ce5cd9640 10 monclient: _renew_subs 2024-03-25T18:08:20.759 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.754+0000 7f0ce5cd9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:08:20.759 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.754+0000 7f0ce5cd9640 1 -- 172.21.15.194:0/3696343388 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce01e3860 con 0x7f0ce0142a20 2024-03-25T18:08:20.759 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.754+0000 7f0ce5cd9640 1 librados: init done 2024-03-25T18:08:20.759 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.755+0000 7f0ce5cd9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:20.759 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.755+0000 7f0ce5cd9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:20.759 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.755+0000 7f0ce5cd9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:08:20.759 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.755+0000 7f0ce5cd9640 1 -- 172.21.15.194:0/3696343388 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ce004eee0 con 0x7f0ce0142a20 2024-03-25T18:08:20.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.758+0000 7f0cc7fff640 1 -- 172.21.15.194:0/3696343388 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cd8001f20 con 0x7f0ce0142a20 2024-03-25T18:08:20.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.758+0000 7f0cc7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:20.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.758+0000 7f0cc7fff640 1 -- 172.21.15.194:0/3696343388 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f0cd8004fb0 con 0x7f0ce0142a20 2024-03-25T18:08:20.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.758+0000 7f0cc7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:20.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.758+0000 7f0cc7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:20.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.758+0000 7f0cc7fff640 10 monclient: dump: 2024-03-25T18:08:20.760 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:20.760 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:20.760 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.760 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:20.760 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:20.761 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:20.761 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:20.761 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:20.761 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:20.761 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:20.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.758+0000 7f0cc7fff640 1 -- 172.21.15.194:0/3696343388 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0cd80054a0 con 0x7f0ce0142a20 2024-03-25T18:08:20.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.758+0000 7f0cddffb640 4 set_mon_vals no callback set 2024-03-25T18:08:20.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.758+0000 7f0cc7fff640 1 -- 172.21.15.194:0/3696343388 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 1939495007 0 0) 0x7f0cd8006b10 con 0x7f0ce0142a20 2024-03-25T18:08:20.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.758+0000 7f0cc7fff640 10 client.4149.objecter ms_dispatch 0x7f0ce0000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T18:08:20.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.758+0000 7f0cc7fff640 3 client.4149.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T18:08:20.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.758+0000 7f0cc7fff640 3 client.4149.objecter handle_osd_map decoding full epoch 7 2024-03-25T18:08:20.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.758+0000 7f0cc7fff640 20 client.4149.objecter dump_active .. 0 homeless 2024-03-25T18:08:20.855 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.855+0000 7f0cc7fff640 1 -- 172.21.15.194:0/3696343388 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0cd80066a0 con 0x7f0ce0142a20 2024-03-25T18:08:20.855 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.855+0000 7f0cc7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:20.855 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.855+0000 7f0cc7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:20.929 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.928+0000 7f0ce5cd9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1", "id": 5}] 2024-03-25T18:08:20.929 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.928+0000 7f0ce5cd9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1", "id": 5}] 2024-03-25T18:08:20.929 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.928+0000 7f0ce5cd9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:08:20.929 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.928+0000 7f0ce5cd9640 1 -- 172.21.15.194:0/3696343388 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1", "id": 5} v 0) v1 -- 0x7f0ce00b3df0 con 0x7f0ce0142a20 2024-03-25T18:08:20.968 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.967+0000 7f0cc7fff640 1 -- 172.21.15.194:0/3696343388 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 2671106593 0 1988412176) 0x7f0cd8006840 con 0x7f0ce0142a20 2024-03-25T18:08:20.968 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.967+0000 7f0cc7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1", "id": 5}] 2024-03-25T18:08:20.968 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.967+0000 7f0cc7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:20.968 INFO:teuthology.orchestra.run.smithi194.stdout:5 2024-03-25T18:08:20.969 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.968+0000 7f0ce5cd9640 10 librados: watch_flush enter 2024-03-25T18:08:20.969 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.968+0000 7f0ce5cd9640 10 librados: watch_flush exit 2024-03-25T18:08:20.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.968+0000 7f0ce5cd9640 20 client.4149.objecter shutdown clearing up homeless session... 2024-03-25T18:08:20.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.968+0000 7f0ce5cd9640 10 client.4149.objecter successfully canceled tick 2024-03-25T18:08:20.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.968+0000 7f0ce5cd9640 10 monclient: shutdown 2024-03-25T18:08:20.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.968+0000 7f0ce5cd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:20.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.968+0000 7f0ce5cd9640 1 -- 172.21.15.194:0/3696343388 >> v1:172.21.15.156:6789/0 conn(0x7f0ce0142a20 legacy=0x7f0ce013a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:20.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.968+0000 7f0ce5cd9640 1 -- 172.21.15.194:0/3696343388 shutdown_connections 2024-03-25T18:08:20.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.968+0000 7f0ce5cd9640 1 -- 172.21.15.194:0/3696343388 >> 172.21.15.194:0/3696343388 conn(0x7f0ce00668e0 msgr2=0x7f0ce00b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:20.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.968+0000 7f0ce5cd9640 1 -- 172.21.15.194:0/3696343388 shutdown_connections 2024-03-25T18:08:20.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.968+0000 7f0ce5cd9640 1 -- 172.21.15.194:0/3696343388 wait complete. 2024-03-25T18:08:20.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:20.969+0000 7f0ce5cd9640 1 librados: shutdown 2024-03-25T18:08:20.980 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new a4b4b316-17b7-4a39-a61f-3004ce88b732 6 2024-03-25T18:08:21.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.080+0000 7f67d1a93640 10 monclient: get_monmap_and_config 2024-03-25T18:08:21.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.080+0000 7f67d1a93640 10 monclient: build_initial_monmap 2024-03-25T18:08:21.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.080+0000 7f67d1a93640 10 monclient: monmap: 2024-03-25T18:08:21.081 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-25T18:08:21.081 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:21.081 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:08:21.081176+0000 2024-03-25T18:08:21.081 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:08:21.081176+0000 2024-03-25T18:08:21.081 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:21.081 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:21.081 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:21.081 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:08:21.081 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:08:21.081 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:21.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.080+0000 7f67d1a93640 1 Processor -- start 2024-03-25T18:08:21.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.080+0000 7f67d1a93640 1 -- start start 2024-03-25T18:08:21.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.080+0000 7f67d1a93640 10 monclient: init 2024-03-25T18:08:21.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.081+0000 7f67d1a93640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:21.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.081+0000 7f67d1a93640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:08:21.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.081+0000 7f67d1a93640 10 monclient(hunting): picked mon.noname-c con 0x7f67cc0a9920 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:21.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.081+0000 7f67d1a93640 10 monclient(hunting): picked mon.noname-a con 0x7f67cc13fce0 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:21.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.081+0000 7f67d1a93640 10 monclient(hunting): picked mon.noname-b con 0x7f67cc142ee0 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:21.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.081+0000 7f67d1a93640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67cc0aa430 con 0x7f67cc13fce0 2024-03-25T18:08:21.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.081+0000 7f67d1a93640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67cc0b27f0 con 0x7f67cc0a9920 2024-03-25T18:08:21.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.081+0000 7f67d1a93640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67cc0b88e0 con 0x7f67cc142ee0 2024-03-25T18:08:21.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.081+0000 7f67d1a93640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:21.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.081+0000 7f67d1a93640 10 monclient(hunting): authenticate will time out at 6971.363285s 2024-03-25T18:08:21.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.082+0000 7f67caffd640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f67cc13fce0 0x7f67cc1427d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.194:56360/0 (socket says 172.21.15.194:56360) 2024-03-25T18:08:21.084 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.082+0000 7f67cbfff640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f67cc142ee0 0x7f67cc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.194:35172/0 (socket says 172.21.15.194:35172) 2024-03-25T18:08:21.084 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.082+0000 7f67caffd640 1 -- 172.21.15.194:0/278069257 learned_addr learned my addr 172.21.15.194:0/278069257 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-25T18:08:21.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.084+0000 7f67ca7fc640 1 -- 172.21.15.194:0/278069257 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1587740436 0 0) 0x7f67bc001110 con 0x7f67cc13fce0 2024-03-25T18:08:21.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.084+0000 7f67ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:21.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.084+0000 7f67ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:21.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.084+0000 7f67ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:21.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.084+0000 7f67ca7fc640 10 monclient(hunting): my global_id is 4152 2024-03-25T18:08:21.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.084+0000 7f67ca7fc640 1 -- 172.21.15.194:0/278069257 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67b4003520 con 0x7f67cc13fce0 2024-03-25T18:08:21.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.084+0000 7f67ca7fc640 1 -- 172.21.15.194:0/278069257 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722055180 0 0) 0x7f67cc0b27f0 con 0x7f67cc0a9920 2024-03-25T18:08:21.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.084+0000 7f67ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:21.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.084+0000 7f67ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:21.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.084+0000 7f67ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:21.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.084+0000 7f67ca7fc640 10 monclient(hunting): my global_id is 4153 2024-03-25T18:08:21.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.084+0000 7f67ca7fc640 1 -- 172.21.15.194:0/278069257 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67bc001110 con 0x7f67cc0a9920 2024-03-25T18:08:21.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.084+0000 7f67ca7fc640 1 -- 172.21.15.194:0/278069257 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288648903 0 0) 0x7f67cc0b88e0 con 0x7f67cc142ee0 2024-03-25T18:08:21.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.084+0000 7f67ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:21.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.084+0000 7f67ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:21.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.084+0000 7f67ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:21.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.084+0000 7f67ca7fc640 10 monclient(hunting): my global_id is 4154 2024-03-25T18:08:21.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.084+0000 7f67ca7fc640 1 -- 172.21.15.194:0/278069257 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67cc0b27f0 con 0x7f67cc142ee0 2024-03-25T18:08:21.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.087+0000 7f67ca7fc640 1 -- 172.21.15.194:0/278069257 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3487924672 0 0) 0x7f67b4003520 con 0x7f67cc13fce0 2024-03-25T18:08:21.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.087+0000 7f67ca7fc640 1 -- 172.21.15.194:0/278069257 >> v1:172.21.15.156:6790/0 conn(0x7f67cc142ee0 legacy=0x7f67cc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:21.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.087+0000 7f67ca7fc640 1 -- 172.21.15.194:0/278069257 >> v1:172.21.15.194:6789/0 conn(0x7f67cc0a9920 legacy=0x7f67cc13f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:21.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.087+0000 7f67ca7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:08:21.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.087+0000 7f67ca7fc640 1 monclient: found mon.noname-a 2024-03-25T18:08:21.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.087+0000 7f67ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:21.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.087+0000 7f67ca7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:08:21.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.087+0000 7f67ca7fc640 1 -- 172.21.15.194:0/278069257 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67cc0b45e0 con 0x7f67cc13fce0 2024-03-25T18:08:21.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.087+0000 7f67ca7fc640 10 monclient: _finish_auth 0 2024-03-25T18:08:21.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.087+0000 7f67ca7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:08:21.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.087+0000 7f67ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:21.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.087+0000 7f67d1a93640 5 monclient: authenticate success, global_id 4152 2024-03-25T18:08:21.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.087+0000 7f67d1a93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:21.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.087+0000 7f67ca7fc640 1 -- 172.21.15.194:0/278069257 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f67bc001640 con 0x7f67cc13fce0 2024-03-25T18:08:21.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.087+0000 7f67ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:21.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.087+0000 7f67ca7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:08:21.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.087+0000 7f67ca7fc640 10 monclient: dump: 2024-03-25T18:08:21.089 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:21.089 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:21.089 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.089 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.089 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:21.090 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:21.090 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:21.090 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:21.090 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:21.090 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:21.090 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.087+0000 7f67d1a93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:21.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.089+0000 7f67ca7fc640 1 -- 172.21.15.194:0/278069257 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67bc001890 con 0x7f67cc13fce0 2024-03-25T18:08:21.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.089+0000 7f67ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:21.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.089+0000 7f67ca7fc640 1 -- 172.21.15.194:0/278069257 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f67bc001c20 con 0x7f67cc13fce0 2024-03-25T18:08:21.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.089+0000 7f67ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:21.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.089+0000 7f67ca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:21.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.089+0000 7f67ca7fc640 10 monclient: dump: 2024-03-25T18:08:21.093 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:21.093 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:21.093 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.093 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.093 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:21.093 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:21.093 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:21.093 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:21.094 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:21.094 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:21.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.089+0000 7f67d1a93640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:21.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.089+0000 7f67d1a93640 4 set_mon_vals no callback set 2024-03-25T18:08:21.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.090+0000 7f67d1a93640 10 monclient: shutdown 2024-03-25T18:08:21.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.090+0000 7f67d1a93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:21.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.090+0000 7f67d1a93640 1 -- 172.21.15.194:0/278069257 >> v1:172.21.15.156:6789/0 conn(0x7f67cc13fce0 legacy=0x7f67cc1427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:21.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.090+0000 7f67d1a93640 1 -- 172.21.15.194:0/278069257 shutdown_connections 2024-03-25T18:08:21.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.090+0000 7f67d1a93640 1 -- 172.21.15.194:0/278069257 >> 172.21.15.194:0/278069257 conn(0x7f67cc0b7df0 msgr2=0x7f67cc0b81d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:21.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.090+0000 7f67d1a93640 1 -- 172.21.15.194:0/278069257 shutdown_connections 2024-03-25T18:08:21.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.090+0000 7f67d1a93640 1 -- 172.21.15.194:0/278069257 wait complete. 2024-03-25T18:08:21.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.090+0000 7f67d1a93640 10 monclient: build_initial_monmap 2024-03-25T18:08:21.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.090+0000 7f67d1a93640 10 monclient: monmap: 2024-03-25T18:08:21.094 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-25T18:08:21.095 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:21.095 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 0.000000 2024-03-25T18:08:21.095 INFO:teuthology.orchestra.run.smithi194.stderr:created 0.000000 2024-03-25T18:08:21.095 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:21.095 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-25T18:08:21.095 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:21.095 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:08:21.095 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:08:21.095 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:21.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.090+0000 7f67d1a93640 1 librados: starting msgr at 2024-03-25T18:08:21.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.090+0000 7f67d1a93640 1 librados: starting objecter 2024-03-25T18:08:21.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.091+0000 7f67d1a93640 1 Processor -- start 2024-03-25T18:08:21.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.091+0000 7f67d1a93640 1 -- start start 2024-03-25T18:08:21.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.091+0000 7f67d1a93640 1 librados: setting wanted keys 2024-03-25T18:08:21.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.091+0000 7f67d1a93640 1 librados: calling monclient init 2024-03-25T18:08:21.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.091+0000 7f67d1a93640 10 monclient: init 2024-03-25T18:08:21.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.091+0000 7f67d1a93640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:21.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.091+0000 7f67d1a93640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:08:21.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.091+0000 7f67d1a93640 10 monclient(hunting): picked mon.noname-a con 0x7f67cc0a9920 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:21.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.091+0000 7f67d1a93640 10 monclient(hunting): picked mon.noname-b con 0x7f67cc13fce0 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:21.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.091+0000 7f67d1a93640 10 monclient(hunting): picked mon.noname-c con 0x7f67cc142ee0 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:21.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.091+0000 7f67d1a93640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67cc0acfc0 con 0x7f67cc0a9920 2024-03-25T18:08:21.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.091+0000 7f67d1a93640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67cc0ad230 con 0x7f67cc13fce0 2024-03-25T18:08:21.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.091+0000 7f67d1a93640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67cc1d7320 con 0x7f67cc142ee0 2024-03-25T18:08:21.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.091+0000 7f67d1a93640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:21.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.091+0000 7f67d1a93640 10 monclient(hunting): authenticate will time out at 6971.373613s 2024-03-25T18:08:21.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.092+0000 7f67cbfff640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f67cc142ee0 0x7f67cc1e2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.194:35180/0 (socket says 172.21.15.194:35180) 2024-03-25T18:08:21.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.092+0000 7f67cbfff640 1 -- 172.21.15.194:0/3661780275 learned_addr learned my addr 172.21.15.194:0/3661780275 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-25T18:08:21.097 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.092+0000 7f67b3fff640 10 client.?.objecter ms_handle_connect 0x7f67cc142ee0 2024-03-25T18:08:21.097 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.092+0000 7f67b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:21.097 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.092+0000 7f67b3fff640 10 client.?.objecter ms_handle_connect 0x7f67cc0a9920 2024-03-25T18:08:21.097 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.092+0000 7f67b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:21.097 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.092+0000 7f67b3fff640 10 client.?.objecter ms_handle_connect 0x7f67cc13fce0 2024-03-25T18:08:21.097 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.092+0000 7f67b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:21.098 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.094+0000 7f67b3fff640 1 -- 172.21.15.194:0/3661780275 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2092418986 0 0) 0x7f67cc0acfc0 con 0x7f67cc0a9920 2024-03-25T18:08:21.098 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.094+0000 7f67b3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:21.098 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.094+0000 7f67b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:21.098 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.094+0000 7f67b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:21.098 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.094+0000 7f67b3fff640 10 monclient(hunting): my global_id is 4155 2024-03-25T18:08:21.098 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.094+0000 7f67b3fff640 1 -- 172.21.15.194:0/3661780275 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67b8004eb0 con 0x7f67cc0a9920 2024-03-25T18:08:21.098 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.094+0000 7f67b3fff640 1 -- 172.21.15.194:0/3661780275 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725441229 0 0) 0x7f67cc0ad230 con 0x7f67cc13fce0 2024-03-25T18:08:21.098 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.094+0000 7f67b3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:21.098 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.094+0000 7f67b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:21.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.094+0000 7f67b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:21.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.094+0000 7f67b3fff640 10 monclient(hunting): my global_id is 4156 2024-03-25T18:08:21.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.094+0000 7f67b3fff640 1 -- 172.21.15.194:0/3661780275 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67cc0acfc0 con 0x7f67cc13fce0 2024-03-25T18:08:21.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.095+0000 7f67b3fff640 1 -- 172.21.15.194:0/3661780275 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177586895 0 0) 0x7f67cc1d7320 con 0x7f67cc142ee0 2024-03-25T18:08:21.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.095+0000 7f67b3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:21.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.095+0000 7f67b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:21.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.095+0000 7f67b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:21.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.095+0000 7f67b3fff640 10 monclient(hunting): my global_id is 4157 2024-03-25T18:08:21.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.095+0000 7f67b3fff640 1 -- 172.21.15.194:0/3661780275 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67cc0ad230 con 0x7f67cc142ee0 2024-03-25T18:08:21.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.099+0000 7f67b3fff640 1 -- 172.21.15.194:0/3661780275 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3554372027 0 0) 0x7f67b8004eb0 con 0x7f67cc0a9920 2024-03-25T18:08:21.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.099+0000 7f67b3fff640 1 -- 172.21.15.194:0/3661780275 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67cc1d7320 con 0x7f67cc0a9920 2024-03-25T18:08:21.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.099+0000 7f67b3fff640 1 -- 172.21.15.194:0/3661780275 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f67c00018b0 con 0x7f67cc0a9920 2024-03-25T18:08:21.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.099+0000 7f67b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:21.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.099+0000 7f67b3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:08:21.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.099+0000 7f67b3fff640 10 monclient(hunting): dump: 2024-03-25T18:08:21.101 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:21.101 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:21.101 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.101 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.101 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:21.101 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:21.101 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:21.101 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:21.101 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:21.101 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:21.101 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.099+0000 7f67b3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:21.101 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.099+0000 7f67b3fff640 1 -- 172.21.15.194:0/3661780275 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1354105941 0 0) 0x7f67cc0acfc0 con 0x7f67cc13fce0 2024-03-25T18:08:21.101 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.099+0000 7f67b3fff640 1 -- 172.21.15.194:0/3661780275 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67b8004eb0 con 0x7f67cc13fce0 2024-03-25T18:08:21.101 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.099+0000 7f67b3fff640 1 -- 172.21.15.194:0/3661780275 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f67bc0047e0 con 0x7f67cc13fce0 2024-03-25T18:08:21.101 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.099+0000 7f67b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:21.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.099+0000 7f67b3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:21.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.099+0000 7f67b3fff640 10 monclient(hunting): dump: 2024-03-25T18:08:21.102 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:21.102 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:21.102 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.102 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.102 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:21.102 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:21.102 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:21.102 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:21.102 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:21.102 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:21.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.102+0000 7f67b3fff640 1 -- 172.21.15.194:0/3661780275 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 574813889 0 0) 0x7f67b8004eb0 con 0x7f67cc13fce0 2024-03-25T18:08:21.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.102+0000 7f67b3fff640 1 -- 172.21.15.194:0/3661780275 >> v1:172.21.15.156:6790/0 conn(0x7f67cc142ee0 legacy=0x7f67cc1e2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:21.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.102+0000 7f67b3fff640 1 -- 172.21.15.194:0/3661780275 >> v1:172.21.15.156:6789/0 conn(0x7f67cc0a9920 legacy=0x7f67cc1d7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:21.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.102+0000 7f67b3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:08:21.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.102+0000 7f67b3fff640 1 monclient: found mon.b 2024-03-25T18:08:21.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.102+0000 7f67b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:21.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.102+0000 7f67b3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:21.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.102+0000 7f67b3fff640 1 -- 172.21.15.194:0/3661780275 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67cc143690 con 0x7f67cc13fce0 2024-03-25T18:08:21.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.102+0000 7f67b3fff640 10 monclient: _finish_auth 0 2024-03-25T18:08:21.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.102+0000 7f67b3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:08:21.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.102+0000 7f67b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:21.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.102+0000 7f67d1a93640 5 monclient: authenticate success, global_id 4156 2024-03-25T18:08:21.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.102+0000 7f67d1a93640 10 monclient: _renew_subs 2024-03-25T18:08:21.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.102+0000 7f67d1a93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:21.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.102+0000 7f67d1a93640 1 -- 172.21.15.194:0/3661780275 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67cc145ed0 con 0x7f67cc13fce0 2024-03-25T18:08:21.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.102+0000 7f67d1a93640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:21.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.103+0000 7f67d1a93640 10 monclient: _renew_subs 2024-03-25T18:08:21.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.103+0000 7f67d1a93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:21.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.103+0000 7f67d1a93640 1 -- 172.21.15.194:0/3661780275 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f67cc146450 con 0x7f67cc13fce0 2024-03-25T18:08:21.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.103+0000 7f67d1a93640 1 librados: init done 2024-03-25T18:08:21.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.103+0000 7f67d1a93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:21.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.103+0000 7f67d1a93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:21.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.103+0000 7f67d1a93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:21.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.103+0000 7f67d1a93640 1 -- 172.21.15.194:0/3661780275 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67cc1d74b0 con 0x7f67cc13fce0 2024-03-25T18:08:21.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.200+0000 7f67b3fff640 1 -- 172.21.15.194:0/3661780275 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67bc001320 con 0x7f67cc13fce0 2024-03-25T18:08:21.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.200+0000 7f67b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:21.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.200+0000 7f67b3fff640 1 -- 172.21.15.194:0/3661780275 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f67bc004d90 con 0x7f67cc13fce0 2024-03-25T18:08:21.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.200+0000 7f67b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:21.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.200+0000 7f67b3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:21.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.200+0000 7f67b3fff640 10 monclient: dump: 2024-03-25T18:08:21.201 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:21.201 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:21.202 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.202 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.202 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:21.202 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:21.202 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:21.202 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:21.202 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:21.202 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:21.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.200+0000 7f67b3fff640 1 -- 172.21.15.194:0/3661780275 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f67bc005300 con 0x7f67cc13fce0 2024-03-25T18:08:21.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.200+0000 7f67b3fff640 1 -- 172.21.15.194:0/3661780275 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 3405198499 0 0) 0x7f67bc006a50 con 0x7f67cc13fce0 2024-03-25T18:08:21.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.200+0000 7f67b3fff640 10 client.4156.objecter ms_dispatch 0x7f67cc000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T18:08:21.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.200+0000 7f67b3fff640 3 client.4156.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T18:08:21.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.200+0000 7f67b3fff640 3 client.4156.objecter handle_osd_map decoding full epoch 8 2024-03-25T18:08:21.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.200+0000 7f67c9ffb640 4 set_mon_vals no callback set 2024-03-25T18:08:21.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.200+0000 7f67b3fff640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-25T18:08:21.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.200+0000 7f67b3fff640 1 -- 172.21.15.194:0/3661780275 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f67bc018de0 con 0x7f67cc13fce0 2024-03-25T18:08:21.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.200+0000 7f67b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:21.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.200+0000 7f67b3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:21.216 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:08:21.216+0000 7f0803836040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:08:21.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.272+0000 7f67d1a93640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a4b4b316-17b7-4a39-a61f-3004ce88b732", "id": 6}] 2024-03-25T18:08:21.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.273+0000 7f67d1a93640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a4b4b316-17b7-4a39-a61f-3004ce88b732", "id": 6}] 2024-03-25T18:08:21.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.273+0000 7f67d1a93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:21.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.273+0000 7f67d1a93640 1 -- 172.21.15.194:0/3661780275 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a4b4b316-17b7-4a39-a61f-3004ce88b732", "id": 6} v 0) v1 -- 0x7f67cc04f730 con 0x7f67cc13fce0 2024-03-25T18:08:21.332 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.331+0000 7f67b3fff640 1 -- 172.21.15.194:0/3661780275 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a4b4b316-17b7-4a39-a61f-3004ce88b732", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 3641518782 0 1113811849) 0x7f67bc006530 con 0x7f67cc13fce0 2024-03-25T18:08:21.332 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.331+0000 7f67b3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a4b4b316-17b7-4a39-a61f-3004ce88b732", "id": 6}] 2024-03-25T18:08:21.332 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.331+0000 7f67b3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:21.332 INFO:teuthology.orchestra.run.smithi194.stdout:6 2024-03-25T18:08:21.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.333+0000 7f67d1a93640 10 librados: watch_flush enter 2024-03-25T18:08:21.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.333+0000 7f67d1a93640 10 librados: watch_flush exit 2024-03-25T18:08:21.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.333+0000 7f67d1a93640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-25T18:08:21.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.333+0000 7f67d1a93640 10 client.4156.objecter successfully canceled tick 2024-03-25T18:08:21.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.333+0000 7f67d1a93640 10 monclient: shutdown 2024-03-25T18:08:21.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.333+0000 7f67d1a93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:21.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.333+0000 7f67d1a93640 1 -- 172.21.15.194:0/3661780275 >> v1:172.21.15.194:6789/0 conn(0x7f67cc13fce0 legacy=0x7f67cc137fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:21.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.333+0000 7f67d1a93640 1 -- 172.21.15.194:0/3661780275 shutdown_connections 2024-03-25T18:08:21.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.333+0000 7f67d1a93640 1 -- 172.21.15.194:0/3661780275 >> 172.21.15.194:0/3661780275 conn(0x7f67cc0b7df0 msgr2=0x7f67cc1393e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:21.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.333+0000 7f67d1a93640 1 -- 172.21.15.194:0/3661780275 shutdown_connections 2024-03-25T18:08:21.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.333+0000 7f67d1a93640 1 -- 172.21.15.194:0/3661780275 wait complete. 2024-03-25T18:08:21.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.333+0000 7f67d1a93640 1 librados: shutdown 2024-03-25T18:08:21.344 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 40b4ef96-668b-4450-bf78-b59d1c4750b4 7 2024-03-25T18:08:21.443 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.443+0000 7fb25146e640 10 monclient: get_monmap_and_config 2024-03-25T18:08:21.443 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.443+0000 7fb25146e640 10 monclient: build_initial_monmap 2024-03-25T18:08:21.444 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.443+0000 7fb25146e640 10 monclient: monmap: 2024-03-25T18:08:21.444 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-25T18:08:21.444 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:21.444 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:08:21.444263+0000 2024-03-25T18:08:21.444 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:08:21.444263+0000 2024-03-25T18:08:21.444 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:21.444 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:21.444 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:21.444 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:08:21.444 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:08:21.444 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:21.446 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.443+0000 7fb25146e640 1 Processor -- start 2024-03-25T18:08:21.446 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.443+0000 7fb25146e640 1 -- start start 2024-03-25T18:08:21.446 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.443+0000 7fb25146e640 10 monclient: init 2024-03-25T18:08:21.446 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.444+0000 7fb25146e640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:21.446 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.444+0000 7fb25146e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:08:21.446 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.444+0000 7fb25146e640 10 monclient(hunting): picked mon.noname-a con 0x7fb24c0668e0 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:21.446 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.444+0000 7fb25146e640 10 monclient(hunting): picked mon.noname-b con 0x7fb24c066cc0 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:21.446 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.444+0000 7fb25146e640 10 monclient(hunting): picked mon.noname-c con 0x7fb24c145cf0 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:21.446 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.444+0000 7fb25146e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24c146340 con 0x7fb24c0668e0 2024-03-25T18:08:21.446 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.444+0000 7fb25146e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24c142dc0 con 0x7fb24c145cf0 2024-03-25T18:08:21.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.444+0000 7fb25146e640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24c06f450 con 0x7fb24c066cc0 2024-03-25T18:08:21.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.444+0000 7fb25146e640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:21.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.444+0000 7fb25146e640 10 monclient(hunting): authenticate will time out at 6971.726242s 2024-03-25T18:08:21.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.445+0000 7fb24affd640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fb24c0668e0 0x7fb24c13f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.194:56376/0 (socket says 172.21.15.194:56376) 2024-03-25T18:08:21.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.445+0000 7fb2427fc640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7fb24c066cc0 0x7fb24c1422b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.194:35190/0 (socket says 172.21.15.194:35190) 2024-03-25T18:08:21.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.445+0000 7fb24affd640 1 -- 172.21.15.194:0/1723819014 learned_addr learned my addr 172.21.15.194:0/1723819014 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-25T18:08:21.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.447+0000 7fb24a7fc640 1 -- 172.21.15.194:0/1723819014 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3687276041 0 0) 0x7fb24c06f450 con 0x7fb24c066cc0 2024-03-25T18:08:21.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.447+0000 7fb24a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:21.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.447+0000 7fb24a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:21.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.447+0000 7fb24a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:21.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.447+0000 7fb24a7fc640 10 monclient(hunting): my global_id is 4160 2024-03-25T18:08:21.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.447+0000 7fb24a7fc640 1 -- 172.21.15.194:0/1723819014 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb238003520 con 0x7fb24c066cc0 2024-03-25T18:08:21.449 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.447+0000 7fb24a7fc640 1 -- 172.21.15.194:0/1723819014 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2686722921 0 0) 0x7fb24c146340 con 0x7fb24c0668e0 2024-03-25T18:08:21.449 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.447+0000 7fb24a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:21.449 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.447+0000 7fb24a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:21.449 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.447+0000 7fb24a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:21.449 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.447+0000 7fb24a7fc640 10 monclient(hunting): my global_id is 4158 2024-03-25T18:08:21.449 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.447+0000 7fb24a7fc640 1 -- 172.21.15.194:0/1723819014 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb24c06f450 con 0x7fb24c0668e0 2024-03-25T18:08:21.449 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.447+0000 7fb24a7fc640 1 -- 172.21.15.194:0/1723819014 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 655882761 0 0) 0x7fb24c142dc0 con 0x7fb24c145cf0 2024-03-25T18:08:21.449 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.447+0000 7fb24a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:21.449 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.447+0000 7fb24a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:21.449 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.447+0000 7fb24a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:21.449 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.447+0000 7fb24a7fc640 10 monclient(hunting): my global_id is 4159 2024-03-25T18:08:21.449 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.447+0000 7fb24a7fc640 1 -- 172.21.15.194:0/1723819014 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb24c146340 con 0x7fb24c145cf0 2024-03-25T18:08:21.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.450+0000 7fb24a7fc640 1 -- 172.21.15.194:0/1723819014 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1863591120 0 0) 0x7fb24c06f450 con 0x7fb24c0668e0 2024-03-25T18:08:21.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.450+0000 7fb24a7fc640 1 -- 172.21.15.194:0/1723819014 >> v1:172.21.15.156:6790/0 conn(0x7fb24c066cc0 legacy=0x7fb24c1422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:21.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.451+0000 7fb24a7fc640 1 -- 172.21.15.194:0/1723819014 >> v1:172.21.15.194:6789/0 conn(0x7fb24c145cf0 legacy=0x7fb24c14dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:21.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.451+0000 7fb24a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:08:21.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.451+0000 7fb24a7fc640 1 monclient: found mon.noname-a 2024-03-25T18:08:21.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.451+0000 7fb24a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:21.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.451+0000 7fb24a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:08:21.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.451+0000 7fb24a7fc640 1 -- 172.21.15.194:0/1723819014 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24c06f730 con 0x7fb24c0668e0 2024-03-25T18:08:21.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.451+0000 7fb24a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:08:21.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.451+0000 7fb24a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:08:21.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.451+0000 7fb24a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:21.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.451+0000 7fb24a7fc640 1 -- 172.21.15.194:0/1723819014 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fb23c001670 con 0x7fb24c0668e0 2024-03-25T18:08:21.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.451+0000 7fb25146e640 5 monclient: authenticate success, global_id 4158 2024-03-25T18:08:21.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.451+0000 7fb25146e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:21.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.451+0000 7fb24a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:21.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.451+0000 7fb24a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:08:21.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.451+0000 7fb24a7fc640 10 monclient: dump: 2024-03-25T18:08:21.453 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:21.453 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:21.453 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.453 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.453 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:21.453 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:21.453 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:21.453 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:21.453 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:21.453 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:21.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.451+0000 7fb25146e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:21.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.452+0000 7fb24a7fc640 1 -- 172.21.15.194:0/1723819014 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb23c0017d0 con 0x7fb24c0668e0 2024-03-25T18:08:21.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.452+0000 7fb24a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:21.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.452+0000 7fb24a7fc640 1 -- 172.21.15.194:0/1723819014 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fb23c001b60 con 0x7fb24c0668e0 2024-03-25T18:08:21.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.452+0000 7fb25146e640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:21.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.452+0000 7fb25146e640 4 set_mon_vals no callback set 2024-03-25T18:08:21.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.452+0000 7fb24a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:21.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.452+0000 7fb24a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:21.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.452+0000 7fb24a7fc640 10 monclient: dump: 2024-03-25T18:08:21.457 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:21.457 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:21.457 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.457 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.457 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:21.457 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:21.457 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:21.457 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:21.457 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:21.457 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:21.457 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.453+0000 7fb25146e640 10 monclient: shutdown 2024-03-25T18:08:21.457 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.453+0000 7fb25146e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:21.457 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.453+0000 7fb25146e640 1 -- 172.21.15.194:0/1723819014 >> v1:172.21.15.156:6789/0 conn(0x7fb24c0668e0 legacy=0x7fb24c13f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:21.457 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.453+0000 7fb25146e640 1 -- 172.21.15.194:0/1723819014 shutdown_connections 2024-03-25T18:08:21.457 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.453+0000 7fb25146e640 1 -- 172.21.15.194:0/1723819014 >> 172.21.15.194:0/1723819014 conn(0x7fb24c06aba0 msgr2=0x7fb24c06afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:21.458 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.453+0000 7fb25146e640 1 -- 172.21.15.194:0/1723819014 shutdown_connections 2024-03-25T18:08:21.458 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.453+0000 7fb25146e640 1 -- 172.21.15.194:0/1723819014 wait complete. 2024-03-25T18:08:21.458 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.454+0000 7fb25146e640 10 monclient: build_initial_monmap 2024-03-25T18:08:21.458 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.454+0000 7fb25146e640 10 monclient: monmap: 2024-03-25T18:08:21.458 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-25T18:08:21.458 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:21.458 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 0.000000 2024-03-25T18:08:21.458 INFO:teuthology.orchestra.run.smithi194.stderr:created 0.000000 2024-03-25T18:08:21.458 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:21.458 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-25T18:08:21.458 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:21.458 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:08:21.458 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:08:21.458 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:21.459 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.454+0000 7fb25146e640 1 librados: starting msgr at 2024-03-25T18:08:21.459 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.454+0000 7fb25146e640 1 librados: starting objecter 2024-03-25T18:08:21.459 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.454+0000 7fb25146e640 1 Processor -- start 2024-03-25T18:08:21.459 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.454+0000 7fb25146e640 1 -- start start 2024-03-25T18:08:21.459 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.454+0000 7fb25146e640 1 librados: setting wanted keys 2024-03-25T18:08:21.459 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.454+0000 7fb25146e640 1 librados: calling monclient init 2024-03-25T18:08:21.459 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.454+0000 7fb25146e640 10 monclient: init 2024-03-25T18:08:21.459 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.455+0000 7fb25146e640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:21.459 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.455+0000 7fb25146e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:08:21.459 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.455+0000 7fb25146e640 10 monclient(hunting): picked mon.noname-b con 0x7fb24c0668e0 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:21.459 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.455+0000 7fb25146e640 10 monclient(hunting): picked mon.noname-a con 0x7fb24c145cf0 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:21.459 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.455+0000 7fb25146e640 10 monclient(hunting): picked mon.noname-c con 0x7fb24c11d630 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:21.459 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.455+0000 7fb25146e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24c1438d0 con 0x7fb24c145cf0 2024-03-25T18:08:21.459 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.455+0000 7fb25146e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24c11dc10 con 0x7fb24c0668e0 2024-03-25T18:08:21.459 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.455+0000 7fb25146e640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24c11e690 con 0x7fb24c11d630 2024-03-25T18:08:21.459 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.455+0000 7fb25146e640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:21.460 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.455+0000 7fb25146e640 10 monclient(hunting): authenticate will time out at 6971.736951s 2024-03-25T18:08:21.460 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.455+0000 7fb2427fc640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fb24c145cf0 0x7fb24c13b560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.194:56380/0 (socket says 172.21.15.194:56380) 2024-03-25T18:08:21.460 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.455+0000 7fb2427fc640 1 -- 172.21.15.194:0/4226642645 learned_addr learned my addr 172.21.15.194:0/4226642645 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-25T18:08:21.460 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.455+0000 7fb243fff640 10 client.?.objecter ms_handle_connect 0x7fb24c145cf0 2024-03-25T18:08:21.460 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.455+0000 7fb243fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:21.460 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.456+0000 7fb243fff640 10 client.?.objecter ms_handle_connect 0x7fb24c0668e0 2024-03-25T18:08:21.460 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.456+0000 7fb243fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:21.460 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.456+0000 7fb243fff640 10 client.?.objecter ms_handle_connect 0x7fb24c11d630 2024-03-25T18:08:21.460 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.456+0000 7fb243fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:21.461 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.457+0000 7fb243fff640 1 -- 172.21.15.194:0/4226642645 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816548560 0 0) 0x7fb24c1438d0 con 0x7fb24c145cf0 2024-03-25T18:08:21.461 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.457+0000 7fb243fff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:21.461 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.457+0000 7fb243fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:21.461 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.457+0000 7fb243fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:21.461 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.457+0000 7fb243fff640 10 monclient(hunting): my global_id is 4161 2024-03-25T18:08:21.461 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.457+0000 7fb243fff640 1 -- 172.21.15.194:0/4226642645 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb22c004eb0 con 0x7fb24c145cf0 2024-03-25T18:08:21.461 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.458+0000 7fb243fff640 1 -- 172.21.15.194:0/4226642645 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 903218230 0 0) 0x7fb24c11dc10 con 0x7fb24c0668e0 2024-03-25T18:08:21.461 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.458+0000 7fb243fff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:21.461 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.458+0000 7fb243fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:21.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.458+0000 7fb243fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:21.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.458+0000 7fb243fff640 10 monclient(hunting): my global_id is 4162 2024-03-25T18:08:21.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.458+0000 7fb243fff640 1 -- 172.21.15.194:0/4226642645 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb24c1438d0 con 0x7fb24c0668e0 2024-03-25T18:08:21.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.458+0000 7fb243fff640 1 -- 172.21.15.194:0/4226642645 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2312297007 0 0) 0x7fb24c11e690 con 0x7fb24c11d630 2024-03-25T18:08:21.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.458+0000 7fb243fff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:21.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.458+0000 7fb243fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:21.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.458+0000 7fb243fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:21.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.458+0000 7fb243fff640 10 monclient(hunting): my global_id is 4163 2024-03-25T18:08:21.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.458+0000 7fb243fff640 1 -- 172.21.15.194:0/4226642645 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb24c11dc10 con 0x7fb24c11d630 2024-03-25T18:08:21.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.462+0000 7fb243fff640 1 -- 172.21.15.194:0/4226642645 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1275238647 0 0) 0x7fb22c004eb0 con 0x7fb24c145cf0 2024-03-25T18:08:21.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.462+0000 7fb243fff640 1 -- 172.21.15.194:0/4226642645 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb24c11e690 con 0x7fb24c145cf0 2024-03-25T18:08:21.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.462+0000 7fb243fff640 1 -- 172.21.15.194:0/4226642645 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fb234001950 con 0x7fb24c145cf0 2024-03-25T18:08:21.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.462+0000 7fb243fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:21.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.462+0000 7fb243fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:08:21.464 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.462+0000 7fb243fff640 10 monclient(hunting): dump: 2024-03-25T18:08:21.464 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:21.464 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:21.464 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.464 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.464 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:21.464 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:21.464 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:21.464 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:21.464 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:21.464 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:21.464 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.462+0000 7fb243fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:21.465 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.463+0000 7fb243fff640 1 -- 172.21.15.194:0/4226642645 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 86134876 0 0) 0x7fb24c1438d0 con 0x7fb24c0668e0 2024-03-25T18:08:21.465 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.463+0000 7fb243fff640 1 -- 172.21.15.194:0/4226642645 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb22c004eb0 con 0x7fb24c0668e0 2024-03-25T18:08:21.465 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.463+0000 7fb243fff640 1 -- 172.21.15.194:0/4226642645 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fb23c001ae0 con 0x7fb24c0668e0 2024-03-25T18:08:21.465 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.463+0000 7fb243fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:21.465 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.463+0000 7fb243fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:21.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.463+0000 7fb243fff640 10 monclient(hunting): dump: 2024-03-25T18:08:21.466 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:21.466 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:21.466 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.466 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.466 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:21.466 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:21.466 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:21.466 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:21.466 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:21.466 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:21.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.463+0000 7fb243fff640 1 -- 172.21.15.194:0/4226642645 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1139891671 0 0) 0x7fb24c11dc10 con 0x7fb24c11d630 2024-03-25T18:08:21.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.463+0000 7fb243fff640 1 -- 172.21.15.194:0/4226642645 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb24c1438d0 con 0x7fb24c11d630 2024-03-25T18:08:21.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.463+0000 7fb243fff640 1 -- 172.21.15.194:0/4226642645 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fb244001c70 con 0x7fb24c11d630 2024-03-25T18:08:21.467 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.463+0000 7fb243fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:21.467 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.463+0000 7fb243fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:21.467 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.463+0000 7fb243fff640 10 monclient(hunting): dump: 2024-03-25T18:08:21.467 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:21.467 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:21.467 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.467 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.467 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:21.467 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:21.467 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:21.467 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:21.467 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:21.467 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:21.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.466+0000 7fb243fff640 1 -- 172.21.15.194:0/4226642645 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3951885534 0 0) 0x7fb22c004eb0 con 0x7fb24c0668e0 2024-03-25T18:08:21.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.466+0000 7fb243fff640 1 -- 172.21.15.194:0/4226642645 >> v1:172.21.15.156:6790/0 conn(0x7fb24c11d630 legacy=0x7fb24c1d77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:21.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.466+0000 7fb243fff640 1 -- 172.21.15.194:0/4226642645 >> v1:172.21.15.156:6789/0 conn(0x7fb24c145cf0 legacy=0x7fb24c13b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:21.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.466+0000 7fb243fff640 10 monclient: _finish_hunting 0 2024-03-25T18:08:21.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.466+0000 7fb243fff640 1 monclient: found mon.b 2024-03-25T18:08:21.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.466+0000 7fb243fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:21.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.466+0000 7fb243fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:21.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.466+0000 7fb243fff640 1 -- 172.21.15.194:0/4226642645 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24c11e970 con 0x7fb24c0668e0 2024-03-25T18:08:21.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.466+0000 7fb243fff640 10 monclient: _finish_auth 0 2024-03-25T18:08:21.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.466+0000 7fb243fff640 10 monclient: _check_auth_tickets 2024-03-25T18:08:21.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.466+0000 7fb243fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:21.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.466+0000 7fb25146e640 5 monclient: authenticate success, global_id 4162 2024-03-25T18:08:21.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.466+0000 7fb25146e640 10 monclient: _renew_subs 2024-03-25T18:08:21.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.466+0000 7fb25146e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:21.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.466+0000 7fb25146e640 1 -- 172.21.15.194:0/4226642645 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb24c13bc70 con 0x7fb24c0668e0 2024-03-25T18:08:21.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.466+0000 7fb25146e640 10 client.4162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:21.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.466+0000 7fb25146e640 10 monclient: _renew_subs 2024-03-25T18:08:21.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.466+0000 7fb25146e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:21.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.466+0000 7fb25146e640 1 -- 172.21.15.194:0/4226642645 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb24c1d7ec0 con 0x7fb24c0668e0 2024-03-25T18:08:21.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.466+0000 7fb25146e640 1 librados: init done 2024-03-25T18:08:21.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.467+0000 7fb25146e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:21.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.467+0000 7fb25146e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:21.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.467+0000 7fb25146e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:21.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.467+0000 7fb25146e640 1 -- 172.21.15.194:0/4226642645 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb24c04f730 con 0x7fb24c0668e0 2024-03-25T18:08:21.505 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:08:21.505+0000 7f0803836040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:08:21.558 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:08:21.557+0000 7f0803836040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:08:21.568 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.567+0000 7fb243fff640 1 -- 172.21.15.194:0/4226642645 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb23c001ca0 con 0x7fb24c0668e0 2024-03-25T18:08:21.568 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.567+0000 7fb243fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:21.568 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.567+0000 7fb243fff640 1 -- 172.21.15.194:0/4226642645 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fb23c004c30 con 0x7fb24c0668e0 2024-03-25T18:08:21.568 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.567+0000 7fb243fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:21.568 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.567+0000 7fb243fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:21.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.567+0000 7fb243fff640 10 monclient: dump: 2024-03-25T18:08:21.569 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-25T18:08:21.569 INFO:teuthology.orchestra.run.smithi194.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:21.569 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.569 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.569 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:21.569 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 3 2024-03-25T18:08:21.569 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:21.569 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:21.569 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:21.569 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:08:21.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.567+0000 7fb243fff640 1 -- 172.21.15.194:0/4226642645 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb23c005120 con 0x7fb24c0668e0 2024-03-25T18:08:21.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.567+0000 7fb249ffb640 4 set_mon_vals no callback set 2024-03-25T18:08:21.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.567+0000 7fb243fff640 1 -- 172.21.15.194:0/4226642645 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 1092695140 0 0) 0x7fb23c006860 con 0x7fb24c0668e0 2024-03-25T18:08:21.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.567+0000 7fb243fff640 10 client.4162.objecter ms_dispatch 0x7fb24c000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T18:08:21.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.567+0000 7fb243fff640 3 client.4162.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T18:08:21.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.567+0000 7fb243fff640 3 client.4162.objecter handle_osd_map decoding full epoch 9 2024-03-25T18:08:21.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.567+0000 7fb243fff640 20 client.4162.objecter dump_active .. 0 homeless 2024-03-25T18:08:21.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.567+0000 7fb243fff640 1 -- 172.21.15.194:0/4226642645 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb23c018e00 con 0x7fb24c0668e0 2024-03-25T18:08:21.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.567+0000 7fb243fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:21.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.567+0000 7fb243fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:21.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.640+0000 7fb25146e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "40b4ef96-668b-4450-bf78-b59d1c4750b4", "id": 7}] 2024-03-25T18:08:21.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.640+0000 7fb25146e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "40b4ef96-668b-4450-bf78-b59d1c4750b4", "id": 7}] 2024-03-25T18:08:21.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.640+0000 7fb25146e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:21.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.640+0000 7fb25146e640 1 -- 172.21.15.194:0/4226642645 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd new", "uuid": "40b4ef96-668b-4450-bf78-b59d1c4750b4", "id": 7} v 0) v1 -- 0x7fb24c1d73d0 con 0x7fb24c0668e0 2024-03-25T18:08:21.675 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-25T18:08:21.674+0000 7f776ee13040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:08:21.679 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:08:21.679+0000 7f0803836040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:08:21.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.705+0000 7fb243fff640 1 -- 172.21.15.194:0/4226642645 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "40b4ef96-668b-4450-bf78-b59d1c4750b4", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 2816643369 0 1371665406) 0x7fb23c006140 con 0x7fb24c0668e0 2024-03-25T18:08:21.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.705+0000 7fb243fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "40b4ef96-668b-4450-bf78-b59d1c4750b4", "id": 7}] 2024-03-25T18:08:21.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.705+0000 7fb243fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:21.706 INFO:teuthology.orchestra.run.smithi194.stdout:7 2024-03-25T18:08:21.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.706+0000 7fb25146e640 10 librados: watch_flush enter 2024-03-25T18:08:21.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.706+0000 7fb25146e640 10 librados: watch_flush exit 2024-03-25T18:08:21.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.706+0000 7fb25146e640 20 client.4162.objecter shutdown clearing up homeless session... 2024-03-25T18:08:21.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.706+0000 7fb25146e640 10 client.4162.objecter successfully canceled tick 2024-03-25T18:08:21.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.706+0000 7fb25146e640 10 monclient: shutdown 2024-03-25T18:08:21.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.706+0000 7fb25146e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:21.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.706+0000 7fb25146e640 1 -- 172.21.15.194:0/4226642645 >> v1:172.21.15.194:6789/0 conn(0x7fb24c0668e0 legacy=0x7fb24c11c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:21.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.706+0000 7fb25146e640 1 -- 172.21.15.194:0/4226642645 shutdown_connections 2024-03-25T18:08:21.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.706+0000 7fb25146e640 1 -- 172.21.15.194:0/4226642645 >> 172.21.15.194:0/4226642645 conn(0x7fb24c06aba0 msgr2=0x7fb24c0aed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:21.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.706+0000 7fb25146e640 1 -- 172.21.15.194:0/4226642645 shutdown_connections 2024-03-25T18:08:21.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.706+0000 7fb25146e640 1 -- 172.21.15.194:0/4226642645 wait complete. 2024-03-25T18:08:21.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-25T18:08:21.706+0000 7fb25146e640 1 librados: shutdown 2024-03-25T18:08:21.716 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:08:21.717 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T18:08:21.717 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T18:08:21.719 INFO:tasks.ceph.osd.0:Started 2024-03-25T18:08:21.719 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:08:21.720 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T18:08:21.720 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T18:08:21.722 INFO:tasks.ceph.osd.1:Started 2024-03-25T18:08:21.722 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:08:21.722 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T18:08:21.722 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T18:08:21.724 INFO:tasks.ceph.osd.2:Started 2024-03-25T18:08:21.724 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:08:21.724 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T18:08:21.724 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T18:08:21.726 INFO:tasks.ceph.osd.3:Started 2024-03-25T18:08:21.727 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:08:21.727 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T18:08:21.727 DEBUG:teuthology.orchestra.run.smithi194:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T18:08:21.729 INFO:tasks.ceph.osd.4:Started 2024-03-25T18:08:21.729 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:08:21.729 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T18:08:21.729 DEBUG:teuthology.orchestra.run.smithi194:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T18:08:21.731 INFO:tasks.ceph.osd.5:Started 2024-03-25T18:08:21.731 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:08:21.731 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T18:08:21.732 DEBUG:teuthology.orchestra.run.smithi194:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T18:08:21.733 INFO:tasks.ceph.osd.6:Started 2024-03-25T18:08:21.733 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:08:21.734 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T18:08:21.734 DEBUG:teuthology.orchestra.run.smithi194:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T18:08:21.736 INFO:tasks.ceph.osd.7:Started 2024-03-25T18:08:21.736 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:21.908 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.907+0000 7f5774d15640 10 monclient: get_monmap_and_config 2024-03-25T18:08:21.908 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.907+0000 7f5774d15640 10 monclient: build_initial_monmap 2024-03-25T18:08:21.909 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.907+0000 7f5774d15640 10 monclient: monmap: 2024-03-25T18:08:21.909 INFO:teuthology.orchestra.run.smithi156.stderr:epoch 0 2024-03-25T18:08:21.909 INFO:teuthology.orchestra.run.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:21.909 INFO:teuthology.orchestra.run.smithi156.stderr:last_changed 2024-03-25T18:08:21.908808+0000 2024-03-25T18:08:21.909 INFO:teuthology.orchestra.run.smithi156.stderr:created 2024-03-25T18:08:21.908808+0000 2024-03-25T18:08:21.909 INFO:teuthology.orchestra.run.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:21.909 INFO:teuthology.orchestra.run.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:21.909 INFO:teuthology.orchestra.run.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:21.909 INFO:teuthology.orchestra.run.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:08:21.909 INFO:teuthology.orchestra.run.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:08:21.909 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-25T18:08:21.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.908+0000 7f5774d15640 1 Processor -- start 2024-03-25T18:08:21.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.908+0000 7f5774d15640 1 -- start start 2024-03-25T18:08:21.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.908+0000 7f5774d15640 10 monclient: init 2024-03-25T18:08:21.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.909+0000 7f5774d15640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:21.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.909+0000 7f5774d15640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:08:21.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.909+0000 7f5774d15640 10 monclient(hunting): picked mon.noname-c con 0x7f57700b4700 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:21.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.909+0000 7f5774d15640 10 monclient(hunting): picked mon.noname-a con 0x7f5770142a20 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:21.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.909+0000 7f5774d15640 10 monclient(hunting): picked mon.noname-b con 0x7f57700a96b0 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:21.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.909+0000 7f5774d15640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57700a9d60 con 0x7f5770142a20 2024-03-25T18:08:21.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.910+0000 7f5774d15640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f577013ff10 con 0x7f57700b4700 2024-03-25T18:08:21.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.910+0000 7f5774d15640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57700aa1f0 con 0x7f57700a96b0 2024-03-25T18:08:21.912 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.910+0000 7f5774d15640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:21.912 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.910+0000 7f5774d15640 10 monclient(hunting): authenticate will time out at 6983.220810s 2024-03-25T18:08:21.912 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.911+0000 7f576ed9d640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f57700a96b0 0x7f577013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:45142/0 (socket says 172.21.15.156:45142) 2024-03-25T18:08:21.912 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.911+0000 7f576ed9d640 1 -- 172.21.15.156:0/2543163228 learned_addr learned my addr 172.21.15.156:0/2543163228 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:08:21.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.915+0000 7f576d59a640 1 -- 172.21.15.156:0/2543163228 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 876929627 0 0) 0x7f57700aa1f0 con 0x7f57700a96b0 2024-03-25T18:08:21.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.915+0000 7f576d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:08:21.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.915+0000 7f576d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:21.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.915+0000 7f576d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:21.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.915+0000 7f576d59a640 10 monclient(hunting): my global_id is 4166 2024-03-25T18:08:21.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.915+0000 7f576d59a640 1 -- 172.21.15.156:0/2543163228 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5758003520 con 0x7f57700a96b0 2024-03-25T18:08:21.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.916+0000 7f576d59a640 1 -- 172.21.15.156:0/2543163228 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149682267 0 0) 0x7f57700a9d60 con 0x7f5770142a20 2024-03-25T18:08:21.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.916+0000 7f576d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:08:21.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.916+0000 7f576d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:21.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.916+0000 7f576d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:21.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.916+0000 7f576d59a640 10 monclient(hunting): my global_id is 4164 2024-03-25T18:08:21.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.916+0000 7f576d59a640 1 -- 172.21.15.156:0/2543163228 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57700aa1f0 con 0x7f5770142a20 2024-03-25T18:08:21.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.923+0000 7f576d59a640 1 -- 172.21.15.156:0/2543163228 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1007701338 0 0) 0x7f57700aa1f0 con 0x7f5770142a20 2024-03-25T18:08:21.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.923+0000 7f576d59a640 1 -- 172.21.15.156:0/2543163228 >> v1:172.21.15.156:6790/0 conn(0x7f57700a96b0 legacy=0x7f577013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:21.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.923+0000 7f576d59a640 1 -- 172.21.15.156:0/2543163228 >> v1:172.21.15.194:6789/0 conn(0x7f57700b4700 legacy=0x7f5770142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:08:21.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.923+0000 7f576d59a640 10 monclient: _finish_hunting 0 2024-03-25T18:08:21.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.923+0000 7f576d59a640 1 monclient: found mon.noname-a 2024-03-25T18:08:21.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.923+0000 7f576d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:21.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.923+0000 7f576d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:08:21.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.923+0000 7f576d59a640 1 -- 172.21.15.156:0/2543163228 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57700b4ae0 con 0x7f5770142a20 2024-03-25T18:08:21.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.923+0000 7f576d59a640 10 monclient: _finish_auth 0 2024-03-25T18:08:21.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.923+0000 7f576d59a640 10 monclient: _check_auth_tickets 2024-03-25T18:08:21.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.923+0000 7f576d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:21.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.923+0000 7f5774d15640 5 monclient: authenticate success, global_id 4164 2024-03-25T18:08:21.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.923+0000 7f5774d15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:21.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.925+0000 7f576d59a640 1 -- 172.21.15.156:0/2543163228 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5764001630 con 0x7f5770142a20 2024-03-25T18:08:21.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.925+0000 7f576d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:21.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.925+0000 7f576d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:08:21.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.925+0000 7f576d59a640 10 monclient: dump: 2024-03-25T18:08:21.927 INFO:teuthology.orchestra.run.smithi156.stderr:epoch 1 2024-03-25T18:08:21.927 INFO:teuthology.orchestra.run.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:21.927 INFO:teuthology.orchestra.run.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.927 INFO:teuthology.orchestra.run.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.927 INFO:teuthology.orchestra.run.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:21.927 INFO:teuthology.orchestra.run.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:21.927 INFO:teuthology.orchestra.run.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:21.927 INFO:teuthology.orchestra.run.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:21.927 INFO:teuthology.orchestra.run.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:21.927 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-25T18:08:21.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.925+0000 7f5774d15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:21.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.927+0000 7f576d59a640 1 -- 172.21.15.156:0/2543163228 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5764001880 con 0x7f5770142a20 2024-03-25T18:08:21.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.927+0000 7f576d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:21.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.927+0000 7f5774d15640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:21.928 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.927+0000 7f5774d15640 4 set_mon_vals no callback set 2024-03-25T18:08:21.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.927+0000 7f576d59a640 1 -- 172.21.15.156:0/2543163228 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5764001c10 con 0x7f5770142a20 2024-03-25T18:08:21.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.927+0000 7f576d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:21.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.927+0000 7f576d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:21.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.927+0000 7f576d59a640 10 monclient: dump: 2024-03-25T18:08:21.929 INFO:teuthology.orchestra.run.smithi156.stderr:epoch 1 2024-03-25T18:08:21.929 INFO:teuthology.orchestra.run.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:21.929 INFO:teuthology.orchestra.run.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.929 INFO:teuthology.orchestra.run.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.929 INFO:teuthology.orchestra.run.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:21.929 INFO:teuthology.orchestra.run.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:21.929 INFO:teuthology.orchestra.run.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:21.929 INFO:teuthology.orchestra.run.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:21.929 INFO:teuthology.orchestra.run.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:21.929 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-25T18:08:21.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.928+0000 7f5774d15640 10 monclient: shutdown 2024-03-25T18:08:21.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.928+0000 7f5774d15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:21.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.928+0000 7f5774d15640 1 -- 172.21.15.156:0/2543163228 >> v1:172.21.15.156:6789/0 conn(0x7f5770142a20 legacy=0x7f577013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:21.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.928+0000 7f5774d15640 1 -- 172.21.15.156:0/2543163228 shutdown_connections 2024-03-25T18:08:21.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.928+0000 7f5774d15640 1 -- 172.21.15.156:0/2543163228 >> 172.21.15.156:0/2543163228 conn(0x7f57700668e0 msgr2=0x7f5770066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:21.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.928+0000 7f5774d15640 1 -- 172.21.15.156:0/2543163228 shutdown_connections 2024-03-25T18:08:21.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.928+0000 7f5774d15640 1 -- 172.21.15.156:0/2543163228 wait complete. 2024-03-25T18:08:21.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.929+0000 7f5774d15640 10 monclient: build_initial_monmap 2024-03-25T18:08:21.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.929+0000 7f5774d15640 10 monclient: monmap: 2024-03-25T18:08:21.933 INFO:teuthology.orchestra.run.smithi156.stderr:epoch 0 2024-03-25T18:08:21.933 INFO:teuthology.orchestra.run.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:21.933 INFO:teuthology.orchestra.run.smithi156.stderr:last_changed 0.000000 2024-03-25T18:08:21.933 INFO:teuthology.orchestra.run.smithi156.stderr:created 0.000000 2024-03-25T18:08:21.933 INFO:teuthology.orchestra.run.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:21.933 INFO:teuthology.orchestra.run.smithi156.stderr:election_strategy: 1 2024-03-25T18:08:21.934 INFO:teuthology.orchestra.run.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:21.934 INFO:teuthology.orchestra.run.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:08:21.934 INFO:teuthology.orchestra.run.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:08:21.934 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-25T18:08:21.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.930+0000 7f5774d15640 1 librados: starting msgr at 2024-03-25T18:08:21.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.930+0000 7f5774d15640 1 librados: starting objecter 2024-03-25T18:08:21.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.930+0000 7f5774d15640 1 Processor -- start 2024-03-25T18:08:21.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.930+0000 7f5774d15640 1 -- start start 2024-03-25T18:08:21.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.930+0000 7f5774d15640 1 librados: setting wanted keys 2024-03-25T18:08:21.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.930+0000 7f5774d15640 1 librados: calling monclient init 2024-03-25T18:08:21.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.930+0000 7f5774d15640 10 monclient: init 2024-03-25T18:08:21.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.931+0000 7f5774d15640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:21.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.931+0000 7f5774d15640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:08:21.934 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.931+0000 7f5774d15640 10 monclient(hunting): picked mon.noname-a con 0x7f57700a96b0 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:21.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.931+0000 7f5774d15640 10 monclient(hunting): picked mon.noname-c con 0x7f57700b4700 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:21.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.931+0000 7f5774d15640 10 monclient(hunting): picked mon.noname-b con 0x7f5770142a20 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:21.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.931+0000 7f5774d15640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57700ae040 con 0x7f57700a96b0 2024-03-25T18:08:21.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.931+0000 7f5774d15640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57700acf10 con 0x7f5770142a20 2024-03-25T18:08:21.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.931+0000 7f5774d15640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57700ae2b0 con 0x7f57700b4700 2024-03-25T18:08:21.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.931+0000 7f5774d15640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:21.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.931+0000 7f5774d15640 10 monclient(hunting): authenticate will time out at 6983.242407s 2024-03-25T18:08:21.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.932+0000 7f576e59c640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f57700a96b0 0x7f5770135700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:55084/0 (socket says 172.21.15.156:55084) 2024-03-25T18:08:21.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.932+0000 7f576e59c640 1 -- 172.21.15.156:0/3670105911 learned_addr learned my addr 172.21.15.156:0/3670105911 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:08:21.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.932+0000 7f575effd640 10 client.?.objecter ms_handle_connect 0x7f57700a96b0 2024-03-25T18:08:21.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.932+0000 7f575effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:21.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.937+0000 7f575effd640 1 -- 172.21.15.156:0/3670105911 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2848441978 0 0) 0x7f57700ae040 con 0x7f57700a96b0 2024-03-25T18:08:21.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.937+0000 7f575effd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:21.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.937+0000 7f575effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:21.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.937+0000 7f575effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:21.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.937+0000 7f575effd640 10 monclient(hunting): my global_id is 4167 2024-03-25T18:08:21.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.937+0000 7f575effd640 1 -- 172.21.15.156:0/3670105911 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5750004eb0 con 0x7f57700a96b0 2024-03-25T18:08:21.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.939+0000 7f575effd640 10 client.?.objecter ms_handle_connect 0x7f57700b4700 2024-03-25T18:08:21.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.939+0000 7f575effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:21.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.940+0000 7f575effd640 10 client.?.objecter ms_handle_connect 0x7f5770142a20 2024-03-25T18:08:21.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.940+0000 7f575effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:21.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.958+0000 7f575effd640 1 -- 172.21.15.156:0/3670105911 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 711278391 0 0) 0x7f5750004eb0 con 0x7f57700a96b0 2024-03-25T18:08:21.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.958+0000 7f575effd640 1 -- 172.21.15.156:0/3670105911 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57700ae040 con 0x7f57700a96b0 2024-03-25T18:08:21.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.958+0000 7f575effd640 1 -- 172.21.15.156:0/3670105911 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5760001960 con 0x7f57700a96b0 2024-03-25T18:08:21.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.958+0000 7f575effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:21.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.958+0000 7f575effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:08:21.959 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.958+0000 7f575effd640 10 monclient(hunting): dump: 2024-03-25T18:08:21.960 INFO:teuthology.orchestra.run.smithi156.stderr:epoch 1 2024-03-25T18:08:21.960 INFO:teuthology.orchestra.run.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:21.960 INFO:teuthology.orchestra.run.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.960 INFO:teuthology.orchestra.run.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:21.960 INFO:teuthology.orchestra.run.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:21.960 INFO:teuthology.orchestra.run.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:21.960 INFO:teuthology.orchestra.run.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:21.960 INFO:teuthology.orchestra.run.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:21.960 INFO:teuthology.orchestra.run.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:21.960 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-25T18:08:21.960 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:21.958+0000 7f575effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:22.053 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-25T18:08:22.052+0000 7f776ee13040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:08:22.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.096+0000 7f575effd640 1 -- 172.21.15.156:0/3670105911 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733160575 0 0) 0x7f57700acf10 con 0x7f5770142a20 2024-03-25T18:08:22.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.096+0000 7f575effd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:22.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.096+0000 7f575effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:22.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.096+0000 7f575effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:22.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.096+0000 7f575effd640 10 monclient(hunting): my global_id is 4165 2024-03-25T18:08:22.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.096+0000 7f575effd640 1 -- 172.21.15.156:0/3670105911 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5750004eb0 con 0x7f5770142a20 2024-03-25T18:08:22.105 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.104+0000 7f575effd640 1 -- 172.21.15.156:0/3670105911 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3021154413 0 0) 0x7f5750004eb0 con 0x7f5770142a20 2024-03-25T18:08:22.105 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.104+0000 7f575effd640 1 -- 172.21.15.156:0/3670105911 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57700acf10 con 0x7f5770142a20 2024-03-25T18:08:22.105 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.104+0000 7f575effd640 1 -- 172.21.15.156:0/3670105911 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5768001cd0 con 0x7f5770142a20 2024-03-25T18:08:22.105 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.104+0000 7f575effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:22.105 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.104+0000 7f575effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:22.105 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.104+0000 7f575effd640 10 monclient(hunting): dump: 2024-03-25T18:08:22.105 INFO:teuthology.orchestra.run.smithi156.stderr:epoch 1 2024-03-25T18:08:22.105 INFO:teuthology.orchestra.run.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:22.105 INFO:teuthology.orchestra.run.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:22.105 INFO:teuthology.orchestra.run.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:22.105 INFO:teuthology.orchestra.run.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:22.105 INFO:teuthology.orchestra.run.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:22.106 INFO:teuthology.orchestra.run.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:22.106 INFO:teuthology.orchestra.run.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:22.106 INFO:teuthology.orchestra.run.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:22.106 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-25T18:08:22.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.111+0000 7f575effd640 1 -- 172.21.15.156:0/3670105911 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 874324913 0 0) 0x7f57700ae040 con 0x7f57700a96b0 2024-03-25T18:08:22.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.111+0000 7f575effd640 1 -- 172.21.15.156:0/3670105911 >> v1:172.21.15.156:6790/0 conn(0x7f57700b4700 legacy=0x7f57700ae6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:22.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.111+0000 7f575effd640 1 -- 172.21.15.156:0/3670105911 >> v1:172.21.15.194:6789/0 conn(0x7f5770142a20 legacy=0x7f57701e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:22.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.111+0000 7f575effd640 10 monclient: _finish_hunting 0 2024-03-25T18:08:22.113 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.111+0000 7f575effd640 1 monclient: found mon.a 2024-03-25T18:08:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.111+0000 7f575effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.111+0000 7f575effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:08:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.111+0000 7f575effd640 1 -- 172.21.15.156:0/3670105911 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f577012fe90 con 0x7f57700a96b0 2024-03-25T18:08:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.111+0000 7f575effd640 10 monclient: _finish_auth 0 2024-03-25T18:08:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.111+0000 7f575effd640 10 monclient: _check_auth_tickets 2024-03-25T18:08:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.111+0000 7f575effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.111+0000 7f5774d15640 5 monclient: authenticate success, global_id 4167 2024-03-25T18:08:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.111+0000 7f5774d15640 10 monclient: _renew_subs 2024-03-25T18:08:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.111+0000 7f5774d15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:08:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.111+0000 7f5774d15640 1 -- 172.21.15.156:0/3670105911 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57700abbe0 con 0x7f57700a96b0 2024-03-25T18:08:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.111+0000 7f5774d15640 10 client.4167.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.111+0000 7f5774d15640 10 monclient: _renew_subs 2024-03-25T18:08:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.111+0000 7f5774d15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:08:22.114 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.111+0000 7f5774d15640 1 -- 172.21.15.156:0/3670105911 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f57700ac9f0 con 0x7f57700a96b0 2024-03-25T18:08:22.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.111+0000 7f5774d15640 1 librados: init done 2024-03-25T18:08:22.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.113+0000 7f575cff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:22.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.113+0000 7f575cff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:22.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.113+0000 7f575cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:08:22.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.113+0000 7f575cff9640 1 -- 172.21.15.156:0/3670105911 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5754005080 con 0x7f57700a96b0 2024-03-25T18:08:22.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.129+0000 7f575effd640 1 -- 172.21.15.156:0/3670105911 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57600035f0 con 0x7f57700a96b0 2024-03-25T18:08:22.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.129+0000 7f575effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:22.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.129+0000 7f575effd640 1 -- 172.21.15.156:0/3670105911 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5760003a00 con 0x7f57700a96b0 2024-03-25T18:08:22.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.129+0000 7f575effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:22.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.129+0000 7f575effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:22.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.129+0000 7f575effd640 10 monclient: dump: 2024-03-25T18:08:22.130 INFO:teuthology.orchestra.run.smithi156.stderr:epoch 1 2024-03-25T18:08:22.130 INFO:teuthology.orchestra.run.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:22.131 INFO:teuthology.orchestra.run.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:22.131 INFO:teuthology.orchestra.run.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:22.131 INFO:teuthology.orchestra.run.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:22.131 INFO:teuthology.orchestra.run.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:22.131 INFO:teuthology.orchestra.run.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:22.131 INFO:teuthology.orchestra.run.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:22.131 INFO:teuthology.orchestra.run.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:22.131 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-25T18:08:22.131 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.129+0000 7f575ffff640 4 set_mon_vals no callback set 2024-03-25T18:08:22.132 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-25T18:08:22.131+0000 7f776ee13040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:08:22.167 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.166+0000 7f575effd640 1 -- 172.21.15.156:0/3670105911 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 3327984428 0 0) 0x7f5760003b60 con 0x7f57700a96b0 2024-03-25T18:08:22.280 INFO:tasks.ceph.mgr.y.smithi156.stderr:2024-03-25T18:08:22.279+0000 7f776ee13040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:08:22.332 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.331+0000 7f575effd640 1 -- 172.21.15.156:0/3670105911 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f57600156e0 con 0x7f57700a96b0 2024-03-25T18:08:22.332 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.331+0000 7f575effd640 10 client.4167.objecter ms_dispatch 0x7f5770000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:22.332 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.331+0000 7f575effd640 3 client.4167.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:22.332 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.331+0000 7f575effd640 3 client.4167.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:22.332 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.331+0000 7f575effd640 20 client.4167.objecter dump_active .. 0 homeless 2024-03-25T18:08:22.332 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.331+0000 7f575effd640 1 -- 172.21.15.156:0/3670105911 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5760014b80 con 0x7f57700a96b0 2024-03-25T18:08:22.332 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.331+0000 7f575effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:22.332 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.331+0000 7f575effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:22.455 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.454+0000 7f575cff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:22.455 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.454+0000 7f575cff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:22.455 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.454+0000 7f575cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:08:22.455 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.454+0000 7f575cff9640 1 -- 172.21.15.156:0/3670105911 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5754005330 con 0x7f57700a96b0 2024-03-25T18:08:22.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.527+0000 7f575effd640 1 -- 172.21.15.156:0/3670105911 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f576001d6c0 con 0x7f57700a96b0 2024-03-25T18:08:22.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.527+0000 7f575effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:22.528 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.527+0000 7f575effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:22.528 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-25T18:08:22.529 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:22.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.528+0000 7f575cff9640 10 librados: watch_flush enter 2024-03-25T18:08:22.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.528+0000 7f575cff9640 10 librados: watch_flush exit 2024-03-25T18:08:22.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.528+0000 7f575cff9640 20 client.4167.objecter shutdown clearing up homeless session... 2024-03-25T18:08:22.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.528+0000 7f575cff9640 10 client.4167.objecter successfully canceled tick 2024-03-25T18:08:22.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.528+0000 7f575cff9640 10 monclient: shutdown 2024-03-25T18:08:22.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.528+0000 7f575cff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:22.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.528+0000 7f575cff9640 1 -- 172.21.15.156:0/3670105911 >> v1:172.21.15.156:6789/0 conn(0x7f57700a96b0 legacy=0x7f5770135700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:22.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.529+0000 7f575cff9640 1 -- 172.21.15.156:0/3670105911 shutdown_connections 2024-03-25T18:08:22.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.529+0000 7f575cff9640 1 -- 172.21.15.156:0/3670105911 >> 172.21.15.156:0/3670105911 conn(0x7f57700668e0 msgr2=0x7f57700b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:22.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.529+0000 7f575cff9640 1 -- 172.21.15.156:0/3670105911 shutdown_connections 2024-03-25T18:08:22.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.529+0000 7f575cff9640 1 -- 172.21.15.156:0/3670105911 wait complete. 2024-03-25T18:08:22.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-25T18:08:22.529+0000 7f575cff9640 1 librados: shutdown 2024-03-25T18:08:22.544 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T18:08:22.544 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T18:08:23.350 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:23.496 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.495+0000 7fcff9a43640 10 monclient: get_monmap_and_config 2024-03-25T18:08:23.496 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.495+0000 7fcff9a43640 10 monclient: build_initial_monmap 2024-03-25T18:08:23.496 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.495+0000 7fcff9a43640 10 monclient: monmap: 2024-03-25T18:08:23.496 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:08:23.497 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:23.497 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:08:23.496551+0000 2024-03-25T18:08:23.497 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:08:23.496551+0000 2024-03-25T18:08:23.497 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:23.497 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:23.497 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:23.497 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:08:23.497 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:08:23.497 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:23.498 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.496+0000 7fcff9a43640 1 Processor -- start 2024-03-25T18:08:23.499 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.496+0000 7fcff9a43640 1 -- start start 2024-03-25T18:08:23.499 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.496+0000 7fcff9a43640 10 monclient: init 2024-03-25T18:08:23.499 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.497+0000 7fcff9a43640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:23.499 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.497+0000 7fcff9a43640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:08:23.499 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.497+0000 7fcff9a43640 10 monclient(hunting): picked mon.noname-b con 0x7fcff40a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:23.499 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.497+0000 7fcff9a43640 10 monclient(hunting): picked mon.noname-c con 0x7fcff413f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:23.499 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.497+0000 7fcff9a43640 10 monclient(hunting): picked mon.noname-a con 0x7fcff41430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:23.499 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.497+0000 7fcff9a43640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff4142520 con 0x7fcff41430c0 2024-03-25T18:08:23.499 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.497+0000 7fcff9a43640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff40b28a0 con 0x7fcff413f710 2024-03-25T18:08:23.499 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.497+0000 7fcff9a43640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff40b2a30 con 0x7fcff40a98f0 2024-03-25T18:08:23.499 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.497+0000 7fcff9a43640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:23.499 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.497+0000 7fcff9a43640 10 monclient(hunting): authenticate will time out at 6984.808328s 2024-03-25T18:08:23.500 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.498+0000 7fcff2ffd640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7fcff40a98f0 0x7fcff40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:45170/0 (socket says 172.21.15.156:45170) 2024-03-25T18:08:23.500 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.498+0000 7fcff2ffd640 1 -- 172.21.15.156:0/1959620322 learned_addr learned my addr 172.21.15.156:0/1959620322 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:08:23.520 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.519+0000 7fcff1ffb640 1 -- 172.21.15.156:0/1959620322 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 699680714 0 0) 0x7fcff40b2a30 con 0x7fcff40a98f0 2024-03-25T18:08:23.520 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.519+0000 7fcff1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:08:23.520 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.519+0000 7fcff1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:23.520 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.519+0000 7fcff1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:23.520 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.519+0000 7fcff1ffb640 10 monclient(hunting): my global_id is 4187 2024-03-25T18:08:23.521 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.519+0000 7fcff1ffb640 1 -- 172.21.15.156:0/1959620322 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfdc003520 con 0x7fcff40a98f0 2024-03-25T18:08:23.525 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.524+0000 7fcff1ffb640 1 -- 172.21.15.156:0/1959620322 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1912587016 0 0) 0x7fcfdc003520 con 0x7fcff40a98f0 2024-03-25T18:08:23.525 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.525+0000 7fcff1ffb640 1 -- 172.21.15.156:0/1959620322 >> v1:172.21.15.194:6789/0 conn(0x7fcff413f710 legacy=0x7fcff41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:08:23.525 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.525+0000 7fcff1ffb640 1 -- 172.21.15.156:0/1959620322 >> v1:172.21.15.156:6789/0 conn(0x7fcff41430c0 legacy=0x7fcff414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:23.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.525+0000 7fcff1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:08:23.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.525+0000 7fcff1ffb640 1 monclient: found mon.noname-b 2024-03-25T18:08:23.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.525+0000 7fcff1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:23.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.525+0000 7fcff1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.156:6790/0 2024-03-25T18:08:23.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.525+0000 7fcff1ffb640 1 -- 172.21.15.156:0/1959620322 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff40b44c0 con 0x7fcff40a98f0 2024-03-25T18:08:23.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.525+0000 7fcff1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:08:23.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.525+0000 7fcff1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:08:23.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.525+0000 7fcff1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:23.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.525+0000 7fcff1ffb640 1 -- 172.21.15.156:0/1959620322 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fcfe4001690 con 0x7fcff40a98f0 2024-03-25T18:08:23.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.525+0000 7fcff1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:23.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.525+0000 7fcff1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:23.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.525+0000 7fcff1ffb640 10 monclient: dump: 2024-03-25T18:08:23.526 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:08:23.526 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:23.526 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:23.527 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:23.527 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:23.527 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:23.527 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:23.527 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:23.527 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:23.527 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:23.527 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.525+0000 7fcff9a43640 5 monclient: authenticate success, global_id 4187 2024-03-25T18:08:23.527 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.525+0000 7fcff9a43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:23.528 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.527+0000 7fcff1ffb640 1 -- 172.21.15.156:0/1959620322 <== mon.2 v1:172.21.15.156:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcfe4001920 con 0x7fcff40a98f0 2024-03-25T18:08:23.528 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.527+0000 7fcff1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:23.528 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.527+0000 7fcff9a43640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:23.528 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.527+0000 7fcff9a43640 4 set_mon_vals no callback set 2024-03-25T18:08:23.529 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.528+0000 7fcff9a43640 10 monclient: shutdown 2024-03-25T18:08:23.529 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.528+0000 7fcff9a43640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:23.529 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.528+0000 7fcff9a43640 1 -- 172.21.15.156:0/1959620322 >> v1:172.21.15.156:6790/0 conn(0x7fcff40a98f0 legacy=0x7fcff40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:23.530 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.528+0000 7fcff9a43640 1 -- 172.21.15.156:0/1959620322 shutdown_connections 2024-03-25T18:08:23.530 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.528+0000 7fcff9a43640 1 -- 172.21.15.156:0/1959620322 >> 172.21.15.156:0/1959620322 conn(0x7fcff40b7ea0 msgr2=0x7fcff40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:23.530 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.528+0000 7fcff9a43640 1 -- 172.21.15.156:0/1959620322 shutdown_connections 2024-03-25T18:08:23.530 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.528+0000 7fcff9a43640 1 -- 172.21.15.156:0/1959620322 wait complete. 2024-03-25T18:08:23.531 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.529+0000 7fcff9a43640 10 monclient: build_initial_monmap 2024-03-25T18:08:23.531 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.530+0000 7fcff9a43640 10 monclient: monmap: 2024-03-25T18:08:23.531 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:08:23.531 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:23.531 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:08:23.531 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:08:23.531 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:23.531 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:08:23.531 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:23.531 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:08:23.532 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:08:23.532 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:23.532 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.530+0000 7fcff9a43640 1 librados: starting msgr at 2024-03-25T18:08:23.532 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.530+0000 7fcff9a43640 1 librados: starting objecter 2024-03-25T18:08:23.532 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.530+0000 7fcff9a43640 1 Processor -- start 2024-03-25T18:08:23.532 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.530+0000 7fcff9a43640 1 -- start start 2024-03-25T18:08:23.532 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.530+0000 7fcff9a43640 1 librados: setting wanted keys 2024-03-25T18:08:23.532 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.530+0000 7fcff9a43640 1 librados: calling monclient init 2024-03-25T18:08:23.532 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.530+0000 7fcff9a43640 10 monclient: init 2024-03-25T18:08:23.533 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.531+0000 7fcff9a43640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:23.533 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.531+0000 7fcff9a43640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:08:23.533 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.531+0000 7fcff9a43640 10 monclient(hunting): picked mon.noname-b con 0x7fcff40a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:23.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.531+0000 7fcff9a43640 10 monclient(hunting): picked mon.noname-c con 0x7fcff413f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:23.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.532+0000 7fcff9a43640 10 monclient(hunting): picked mon.noname-a con 0x7fcff41430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:23.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.532+0000 7fcff9a43640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff41d7590 con 0x7fcff41430c0 2024-03-25T18:08:23.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.532+0000 7fcff9a43640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff41d51f0 con 0x7fcff40a98f0 2024-03-25T18:08:23.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.532+0000 7fcff9a43640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff41d8460 con 0x7fcff413f710 2024-03-25T18:08:23.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.532+0000 7fcff9a43640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:23.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.532+0000 7fcff9a43640 10 monclient(hunting): authenticate will time out at 6984.843287s 2024-03-25T18:08:23.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.532+0000 7fcff27fc640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7fcff413f710 0x7fcff4138490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:45180/0 (socket says 172.21.15.156:45180) 2024-03-25T18:08:23.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.532+0000 7fcff27fc640 1 -- 172.21.15.156:0/2638594987 learned_addr learned my addr 172.21.15.156:0/2638594987 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:08:23.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.533+0000 7fcfdbfff640 10 client.?.objecter ms_handle_connect 0x7fcff413f710 2024-03-25T18:08:23.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.533+0000 7fcfdbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:23.540 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.539+0000 7fcfdbfff640 10 client.?.objecter ms_handle_connect 0x7fcff40a98f0 2024-03-25T18:08:23.540 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.539+0000 7fcfdbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:23.541 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.540+0000 7fcfdbfff640 10 client.?.objecter ms_handle_connect 0x7fcff41430c0 2024-03-25T18:08:23.541 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.540+0000 7fcfdbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:23.543 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.542+0000 7fcfdbfff640 1 -- 172.21.15.156:0/2638594987 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2377036043 0 0) 0x7fcff41d51f0 con 0x7fcff40a98f0 2024-03-25T18:08:23.543 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.542+0000 7fcfdbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:23.543 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.542+0000 7fcfdbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:23.543 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.542+0000 7fcfdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:23.543 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.542+0000 7fcfdbfff640 10 monclient(hunting): my global_id is 4174 2024-03-25T18:08:23.543 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.542+0000 7fcfdbfff640 1 -- 172.21.15.156:0/2638594987 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfe0004eb0 con 0x7fcff40a98f0 2024-03-25T18:08:23.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.546+0000 7fcfdbfff640 1 -- 172.21.15.156:0/2638594987 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194190630 0 0) 0x7fcff41d7590 con 0x7fcff41430c0 2024-03-25T18:08:23.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.546+0000 7fcfdbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:23.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.546+0000 7fcfdbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:23.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.546+0000 7fcfdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:23.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.546+0000 7fcfdbfff640 10 monclient(hunting): my global_id is 4176 2024-03-25T18:08:23.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.546+0000 7fcfdbfff640 1 -- 172.21.15.156:0/2638594987 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcff41d51f0 con 0x7fcff41430c0 2024-03-25T18:08:23.551 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.550+0000 7fcfdbfff640 1 -- 172.21.15.156:0/2638594987 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 893024157 0 0) 0x7fcfe0004eb0 con 0x7fcff40a98f0 2024-03-25T18:08:23.551 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.550+0000 7fcfdbfff640 1 -- 172.21.15.156:0/2638594987 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcff41d7590 con 0x7fcff40a98f0 2024-03-25T18:08:23.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.553+0000 7fcfdbfff640 1 -- 172.21.15.156:0/2638594987 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fcfe4004ae0 con 0x7fcff40a98f0 2024-03-25T18:08:23.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.553+0000 7fcfdbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:23.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.553+0000 7fcfdbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:23.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.553+0000 7fcfdbfff640 10 monclient(hunting): dump: 2024-03-25T18:08:23.554 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:08:23.554 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:23.555 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:23.555 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:23.555 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:23.555 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:23.555 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:23.555 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:23.555 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:23.555 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:23.555 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.553+0000 7fcfdbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:23.556 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.555+0000 7fcfdbfff640 1 -- 172.21.15.156:0/2638594987 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3661729222 0 0) 0x7fcff41d51f0 con 0x7fcff41430c0 2024-03-25T18:08:23.556 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.555+0000 7fcfdbfff640 1 -- 172.21.15.156:0/2638594987 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfe0004eb0 con 0x7fcff41430c0 2024-03-25T18:08:23.556 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.555+0000 7fcfdbfff640 1 -- 172.21.15.156:0/2638594987 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fcfec001b10 con 0x7fcff41430c0 2024-03-25T18:08:23.556 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.555+0000 7fcfdbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:23.556 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.555+0000 7fcfdbfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:23.556 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.555+0000 7fcfdbfff640 10 monclient(hunting): dump: 2024-03-25T18:08:23.556 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:08:23.556 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:23.556 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:23.557 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:23.557 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:23.557 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:23.557 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:23.557 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:23.557 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:23.557 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:23.563 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.562+0000 7fcfdbfff640 1 -- 172.21.15.156:0/2638594987 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2941359360 0 0) 0x7fcfe0004eb0 con 0x7fcff41430c0 2024-03-25T18:08:23.563 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.562+0000 7fcfdbfff640 1 -- 172.21.15.156:0/2638594987 >> v1:172.21.15.156:6790/0 conn(0x7fcff413f710 legacy=0x7fcff4138490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:23.563 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.562+0000 7fcfdbfff640 1 -- 172.21.15.156:0/2638594987 >> v1:172.21.15.194:6789/0 conn(0x7fcff40a98f0 legacy=0x7fcff41d7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:23.563 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.562+0000 7fcfdbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:08:23.563 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.562+0000 7fcfdbfff640 1 monclient: found mon.a 2024-03-25T18:08:23.563 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.562+0000 7fcfdbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:23.563 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.562+0000 7fcfdbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:08:23.563 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.562+0000 7fcfdbfff640 1 -- 172.21.15.156:0/2638594987 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff41d7830 con 0x7fcff41430c0 2024-03-25T18:08:23.563 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.562+0000 7fcfdbfff640 10 monclient: _finish_auth 0 2024-03-25T18:08:23.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.562+0000 7fcfdbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:08:23.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.562+0000 7fcfdbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:23.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.562+0000 7fcff9a43640 5 monclient: authenticate success, global_id 4176 2024-03-25T18:08:23.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.562+0000 7fcff9a43640 10 monclient: _renew_subs 2024-03-25T18:08:23.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.562+0000 7fcff9a43640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:08:23.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.562+0000 7fcff9a43640 1 -- 172.21.15.156:0/2638594987 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcff4138ba0 con 0x7fcff41430c0 2024-03-25T18:08:23.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.562+0000 7fcff9a43640 10 client.4176.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:23.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.562+0000 7fcff9a43640 10 monclient: _renew_subs 2024-03-25T18:08:23.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.562+0000 7fcff9a43640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:08:23.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.562+0000 7fcff9a43640 1 -- 172.21.15.156:0/2638594987 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcff41379f0 con 0x7fcff41430c0 2024-03-25T18:08:23.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.562+0000 7fcff9a43640 1 librados: init done 2024-03-25T18:08:23.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.563+0000 7fcff9a43640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:23.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.563+0000 7fcff9a43640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:23.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.563+0000 7fcff9a43640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:08:23.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.563+0000 7fcff9a43640 1 -- 172.21.15.156:0/2638594987 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcff40627b0 con 0x7fcff41430c0 2024-03-25T18:08:23.749 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.748+0000 7fcfdbfff640 1 -- 172.21.15.156:0/2638594987 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcfec002000 con 0x7fcff41430c0 2024-03-25T18:08:23.749 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.748+0000 7fcfdbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:23.749 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.748+0000 7fcfdbfff640 1 -- 172.21.15.156:0/2638594987 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fcfec003b30 con 0x7fcff41430c0 2024-03-25T18:08:23.749 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.748+0000 7fcfdbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:23.750 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.748+0000 7fcfdbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:23.750 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.748+0000 7fcfdbfff640 10 monclient: dump: 2024-03-25T18:08:23.750 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:08:23.750 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:23.750 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:23.750 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:23.750 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:23.750 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:23.750 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:23.750 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:23.750 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:23.750 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:23.751 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.748+0000 7fcff17fa640 4 set_mon_vals no callback set 2024-03-25T18:08:23.751 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.749+0000 7fcfdbfff640 1 -- 172.21.15.156:0/2638594987 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 3327984428 0 0) 0x7fcfec003d50 con 0x7fcff41430c0 2024-03-25T18:08:23.755 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.754+0000 7fcfdbfff640 1 -- 172.21.15.156:0/2638594987 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7fcfec014680 con 0x7fcff41430c0 2024-03-25T18:08:23.755 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.754+0000 7fcfdbfff640 10 client.4176.objecter ms_dispatch 0x7fcff4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:23.755 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.754+0000 7fcfdbfff640 3 client.4176.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:23.756 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.754+0000 7fcfdbfff640 3 client.4176.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:23.756 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.754+0000 7fcfdbfff640 20 client.4176.objecter dump_active .. 0 homeless 2024-03-25T18:08:23.756 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.754+0000 7fcfdbfff640 1 -- 172.21.15.156:0/2638594987 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fcfec013b20 con 0x7fcff41430c0 2024-03-25T18:08:23.756 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.754+0000 7fcfdbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:23.756 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.754+0000 7fcfdbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:23.879 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.879+0000 7fcff9a43640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:23.880 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.879+0000 7fcff9a43640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:23.880 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.879+0000 7fcff9a43640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:08:23.880 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.879+0000 7fcff9a43640 1 -- 172.21.15.156:0/2638594987 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcff4141a00 con 0x7fcff41430c0 2024-03-25T18:08:23.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.994+0000 7fcfdbfff640 1 -- 172.21.15.156:0/2638594987 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 1772410519 0 0) 0x7fcfec0043e0 con 0x7fcff41430c0 2024-03-25T18:08:23.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.995+0000 7fcff2ffd640 10 monclient: get_auth_request con 0x7fcfe006e340 auth_method 0 2024-03-25T18:08:23.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:23.995+0000 7fcfdbfff640 10 client.4176.objecter ms_handle_connect 0x7fcfe006e340 2024-03-25T18:08:24.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:24.071+0000 7fcfdbfff640 1 -- 172.21.15.156:0/2638594987 <== mon.0 v1:172.21.15.156:6789/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7fcfec008df0 con 0x7fcff41430c0 2024-03-25T18:08:24.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:24.071+0000 7fcfdbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:24.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:24.071+0000 7fcfdbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:24.072 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:08:24.072 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:24.074 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:24.073+0000 7fcff9a43640 10 librados: watch_flush enter 2024-03-25T18:08:24.074 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:24.073+0000 7fcff9a43640 10 librados: watch_flush exit 2024-03-25T18:08:24.074 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:24.073+0000 7fcff9a43640 20 client.4176.objecter shutdown clearing up homeless session... 2024-03-25T18:08:24.074 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:24.073+0000 7fcff9a43640 10 client.4176.objecter successfully canceled tick 2024-03-25T18:08:24.074 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:24.073+0000 7fcff9a43640 1 -- 172.21.15.156:0/2638594987 >> v1:172.21.15.194:6800/2946427713 conn(0x7fcfe006e340 legacy=0x7fcfe0070730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:24.074 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:24.073+0000 7fcff9a43640 10 monclient: shutdown 2024-03-25T18:08:24.074 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:24.073+0000 7fcff9a43640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:24.074 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:24.073+0000 7fcff9a43640 1 -- 172.21.15.156:0/2638594987 >> v1:172.21.15.156:6789/0 conn(0x7fcff41430c0 legacy=0x7fcff41e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:24.074 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:24.073+0000 7fcff9a43640 1 -- 172.21.15.156:0/2638594987 shutdown_connections 2024-03-25T18:08:24.075 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:24.073+0000 7fcff9a43640 1 -- 172.21.15.156:0/2638594987 >> 172.21.15.156:0/2638594987 conn(0x7fcff40b7ea0 msgr2=0x7fcff4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:24.075 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:24.073+0000 7fcff9a43640 1 -- 172.21.15.156:0/2638594987 shutdown_connections 2024-03-25T18:08:24.075 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:24.073+0000 7fcff9a43640 1 -- 172.21.15.156:0/2638594987 wait complete. 2024-03-25T18:08:24.075 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:24.073+0000 7fcff9a43640 1 librados: shutdown 2024-03-25T18:08:24.092 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:24.110 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:08:24.110+0000 7f07a4610640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T18:08:24.326 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:08:24.325+0000 7f07a4610640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T18:08:28.582 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:06.788 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:28.670 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:06.876 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:28.767 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:06.972 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:28.802 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:07.009 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:28.804 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:07.004 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:28.830 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:07.035 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:28.862 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:07.070 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:28.864 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:07.064 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:29.173 INFO:tasks.ceph.osd.3.smithi156.stderr:==00:00:00:07.353 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:29.208 INFO:tasks.ceph.osd.2.smithi156.stderr:==00:00:00:07.398 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:29.235 INFO:tasks.ceph.osd.3.smithi156.stderr:==00:00:00:07.416 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:29.240 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-03-25T18:08:29.234+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:08:29.267 INFO:tasks.ceph.osd.2.smithi156.stderr:==00:00:00:07.458 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:29.368 INFO:tasks.ceph.osd.1.smithi156.stderr:==00:00:00:07.573 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:29.388 INFO:tasks.ceph.osd.6.smithi194.stderr:2024-03-25T18:08:29.381+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:08:29.414 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-03-25T18:08:29.408+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:08:29.427 INFO:tasks.ceph.osd.1.smithi156.stderr:==00:00:00:07.632 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:29.441 INFO:tasks.ceph.osd.4.smithi194.stderr:2024-03-25T18:08:29.435+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:08:29.446 INFO:tasks.ceph.osd.0.smithi156.stderr:==00:00:00:07.638 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:29.508 INFO:tasks.ceph.osd.0.smithi156.stderr:==00:00:00:07.699 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:29.715 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:07.921 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:29.779 INFO:tasks.ceph.osd.3.smithi156.stderr:2024-03-25T18:08:29.773+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:08:29.841 INFO:tasks.ceph.osd.2.smithi156.stderr:2024-03-25T18:08:29.835+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:08:29.896 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:08.101 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:29.898 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:08.106 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:29.920 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:08.120 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:29.986 INFO:tasks.ceph.osd.1.smithi156.stderr:2024-03-25T18:08:29.979+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:08:30.082 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:08.288 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:30.083 INFO:tasks.ceph.osd.0.smithi156.stderr:2024-03-25T18:08:30.076+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:08:30.264 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:08.469 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:30.266 INFO:tasks.ceph.osd.3.smithi156.stderr:==00:00:00:08.447 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:30.268 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:08.476 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:30.296 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:08.496 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:30.312 INFO:tasks.ceph.osd.2.smithi156.stderr:==00:00:00:08.502 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:30.393 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:08.599 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:30.477 INFO:tasks.ceph.osd.1.smithi156.stderr:==00:00:00:08.682 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:30.557 INFO:tasks.ceph.osd.0.smithi156.stderr:==00:00:00:08.749 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:30.587 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:08.794 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:30.625 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:08.830 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:30.637 INFO:tasks.ceph.osd.3.smithi156.stderr:==00:00:00:08.819 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:30.654 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:08.854 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:30.662 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:08.868 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:30.682 INFO:tasks.ceph.osd.2.smithi156.stderr:==00:00:00:08.872 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:30.844 INFO:tasks.ceph.osd.1.smithi156.stderr:==00:00:00:09.049 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:30.852 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:09.059 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:30.898 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:30.912 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:09.117 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:30.937 INFO:tasks.ceph.osd.0.smithi156.stderr:==00:00:00:09.126 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:30.938 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:09.144 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:30.955 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:09.155 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:30.987 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.986+0000 7f03b6ffe640 10 monclient: get_monmap_and_config 2024-03-25T18:08:30.987 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.986+0000 7f03b6ffe640 10 monclient: build_initial_monmap 2024-03-25T18:08:30.987 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.986+0000 7f03b6ffe640 10 monclient: monmap: 2024-03-25T18:08:30.987 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:08:30.987 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:30.988 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:08:30.987963+0000 2024-03-25T18:08:30.988 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:08:30.987963+0000 2024-03-25T18:08:30.988 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:30.988 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:30.988 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:30.988 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:08:30.988 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:08:30.988 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:30.989 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.987+0000 7f03b6ffe640 1 Processor -- start 2024-03-25T18:08:30.989 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.987+0000 7f03b6ffe640 1 -- start start 2024-03-25T18:08:30.989 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.987+0000 7f03b6ffe640 10 monclient: init 2024-03-25T18:08:30.989 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.988+0000 7f03b6ffe640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:30.989 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.988+0000 7f03b6ffe640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:08:30.989 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.988+0000 7f03b6ffe640 10 monclient(hunting): picked mon.noname-b con 0x7f03b00ae6e0 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:30.989 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.988+0000 7f03b6ffe640 10 monclient(hunting): picked mon.noname-a con 0x7f03b006f080 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:30.989 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.988+0000 7f03b6ffe640 10 monclient(hunting): picked mon.noname-c con 0x7f03b00704e0 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:30.989 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.988+0000 7f03b6ffe640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03b00718d0 con 0x7f03b006f080 2024-03-25T18:08:30.990 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.988+0000 7f03b6ffe640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03b006fa30 con 0x7f03b00704e0 2024-03-25T18:08:30.990 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.988+0000 7f03b6ffe640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03b0070090 con 0x7f03b00ae6e0 2024-03-25T18:08:30.990 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.988+0000 7f03b6ffe640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:30.990 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.988+0000 7f03b6ffe640 10 monclient(hunting): authenticate will time out at 6992.299099s 2024-03-25T18:08:30.990 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.988+0000 7f03b4d9a640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f03b00ae6e0 0x7f03b013c320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:45214/0 (socket says 172.21.15.156:45214) 2024-03-25T18:08:30.990 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.988+0000 7f03b4d9a640 1 -- 172.21.15.156:0/2402884539 learned_addr learned my addr 172.21.15.156:0/2402884539 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:08:30.992 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.991+0000 7f03af7fe640 1 -- 172.21.15.156:0/2402884539 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3215247242 0 0) 0x7f03b006fa30 con 0x7f03b00704e0 2024-03-25T18:08:30.993 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.991+0000 7f03af7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:30.993 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.991+0000 7f03af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:30.993 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.991+0000 7f03af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:30.993 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.991+0000 7f03af7fe640 10 monclient(hunting): my global_id is 4201 2024-03-25T18:08:30.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.991+0000 7f03af7fe640 1 -- 172.21.15.156:0/2402884539 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03a0003520 con 0x7f03b00704e0 2024-03-25T18:08:30.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.991+0000 7f03af7fe640 1 -- 172.21.15.156:0/2402884539 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 433254221 0 0) 0x7f03b00718d0 con 0x7f03b006f080 2024-03-25T18:08:30.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.991+0000 7f03af7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:30.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.991+0000 7f03af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:30.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.991+0000 7f03af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:30.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.991+0000 7f03af7fe640 10 monclient(hunting): my global_id is 4203 2024-03-25T18:08:30.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.991+0000 7f03af7fe640 1 -- 172.21.15.156:0/2402884539 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03b006fa30 con 0x7f03b006f080 2024-03-25T18:08:30.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.992+0000 7f03af7fe640 1 -- 172.21.15.156:0/2402884539 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056285854 0 0) 0x7f03b0070090 con 0x7f03b00ae6e0 2024-03-25T18:08:30.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.992+0000 7f03af7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:30.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.992+0000 7f03af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:30.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.992+0000 7f03af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:30.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.992+0000 7f03af7fe640 10 monclient(hunting): my global_id is 4214 2024-03-25T18:08:30.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.992+0000 7f03af7fe640 1 -- 172.21.15.156:0/2402884539 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03b00718d0 con 0x7f03b00ae6e0 2024-03-25T18:08:30.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.994+0000 7f03af7fe640 1 -- 172.21.15.156:0/2402884539 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 655465159 0 0) 0x7f03a0003520 con 0x7f03b00704e0 2024-03-25T18:08:30.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.994+0000 7f03af7fe640 1 -- 172.21.15.156:0/2402884539 >> v1:172.21.15.156:6790/0 conn(0x7f03b00ae6e0 legacy=0x7f03b013c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:30.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.994+0000 7f03af7fe640 1 -- 172.21.15.156:0/2402884539 >> v1:172.21.15.156:6789/0 conn(0x7f03b006f080 legacy=0x7f03b0071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:30.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.994+0000 7f03af7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:08:30.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.994+0000 7f03af7fe640 1 monclient: found mon.noname-c 2024-03-25T18:08:30.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.994+0000 7f03af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:30.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.994+0000 7f03af7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:08:30.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.994+0000 7f03af7fe640 1 -- 172.21.15.156:0/2402884539 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b006f510 con 0x7f03b00704e0 2024-03-25T18:08:30.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.994+0000 7f03af7fe640 10 monclient: _finish_auth 0 2024-03-25T18:08:30.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.994+0000 7f03af7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:08:30.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.994+0000 7f03af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:30.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.994+0000 7f03af7fe640 1 -- 172.21.15.156:0/2402884539 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f03a8001a10 con 0x7f03b00704e0 2024-03-25T18:08:30.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.994+0000 7f03af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:30.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.994+0000 7f03af7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:30.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.994+0000 7f03af7fe640 10 monclient: dump: 2024-03-25T18:08:30.996 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:08:30.996 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:30.996 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:30.996 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:30.997 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:30.997 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:30.997 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:30.997 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:30.997 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:30.997 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:30.997 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.994+0000 7f03b6ffe640 5 monclient: authenticate success, global_id 4201 2024-03-25T18:08:30.997 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.994+0000 7f03b6ffe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:30.998 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.995+0000 7f03af7fe640 1 -- 172.21.15.156:0/2402884539 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03a8001b70 con 0x7f03b00704e0 2024-03-25T18:08:30.999 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.995+0000 7f03af7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:30.999 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.995+0000 7f03af7fe640 1 -- 172.21.15.156:0/2402884539 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f03a8001f80 con 0x7f03b00704e0 2024-03-25T18:08:30.999 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.995+0000 7f03b6ffe640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:30.999 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.995+0000 7f03b6ffe640 4 set_mon_vals no callback set 2024-03-25T18:08:30.999 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.995+0000 7f03af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:30.999 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.996+0000 7f03af7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:30.999 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.996+0000 7f03af7fe640 10 monclient: dump: 2024-03-25T18:08:30.999 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:08:30.999 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:30.999 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:30.999 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:30.999 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:30.999 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:31.000 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:31.000 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:31.000 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:31.000 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:31.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.996+0000 7f03b6ffe640 10 monclient: shutdown 2024-03-25T18:08:31.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.996+0000 7f03b6ffe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:31.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.996+0000 7f03b6ffe640 1 -- 172.21.15.156:0/2402884539 >> v1:172.21.15.194:6789/0 conn(0x7f03b00704e0 legacy=0x7f03b014a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:31.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.996+0000 7f03b6ffe640 1 -- 172.21.15.156:0/2402884539 shutdown_connections 2024-03-25T18:08:31.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.996+0000 7f03b6ffe640 1 -- 172.21.15.156:0/2402884539 >> 172.21.15.156:0/2402884539 conn(0x7f03b006b630 msgr2=0x7f03b00b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:31.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.996+0000 7f03b6ffe640 1 -- 172.21.15.156:0/2402884539 shutdown_connections 2024-03-25T18:08:31.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.996+0000 7f03b6ffe640 1 -- 172.21.15.156:0/2402884539 wait complete. 2024-03-25T18:08:31.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.997+0000 7f03b6ffe640 10 monclient: build_initial_monmap 2024-03-25T18:08:31.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.997+0000 7f03b6ffe640 10 monclient: monmap: 2024-03-25T18:08:31.000 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:08:31.000 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:31.000 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:08:31.001 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:08:31.001 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:31.001 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:08:31.001 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:31.001 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:08:31.001 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:08:31.001 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:31.001 INFO:tasks.ceph.osd.2.smithi156.stderr:==00:00:00:09.189 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:31.003 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.998+0000 7f03b6ffe640 1 librados: starting msgr at 2024-03-25T18:08:31.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.998+0000 7f03b6ffe640 1 librados: starting objecter 2024-03-25T18:08:31.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.998+0000 7f03b6ffe640 1 Processor -- start 2024-03-25T18:08:31.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.998+0000 7f03b6ffe640 1 -- start start 2024-03-25T18:08:31.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.998+0000 7f03b6ffe640 1 librados: setting wanted keys 2024-03-25T18:08:31.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.998+0000 7f03b6ffe640 1 librados: calling monclient init 2024-03-25T18:08:31.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.998+0000 7f03b6ffe640 10 monclient: init 2024-03-25T18:08:31.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.999+0000 7f03b6ffe640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:31.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.999+0000 7f03b6ffe640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:08:31.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.999+0000 7f03b6ffe640 10 monclient(hunting): picked mon.noname-c con 0x7f03b00704e0 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:31.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.999+0000 7f03b6ffe640 10 monclient(hunting): picked mon.noname-a con 0x7f03b00ae6e0 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:31.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.999+0000 7f03b6ffe640 10 monclient(hunting): picked mon.noname-b con 0x7f03b0142bf0 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:31.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.999+0000 7f03b6ffe640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03b00a9470 con 0x7f03b00ae6e0 2024-03-25T18:08:31.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.999+0000 7f03b6ffe640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03b00aaf10 con 0x7f03b0142bf0 2024-03-25T18:08:31.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.999+0000 7f03b6ffe640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03b01d73f0 con 0x7f03b00704e0 2024-03-25T18:08:31.005 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.999+0000 7f03b6ffe640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:31.005 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.999+0000 7f03b6ffe640 10 monclient(hunting): authenticate will time out at 6992.310399s 2024-03-25T18:08:31.005 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.999+0000 7f03b4d9a640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f03b00704e0 0x7f03b01d6ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:45230/0 (socket says 172.21.15.156:45230) 2024-03-25T18:08:31.005 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:30.999+0000 7f03b4d9a640 1 -- 172.21.15.156:0/1573043938 learned_addr learned my addr 172.21.15.156:0/1573043938 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:08:31.005 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.000+0000 7f03ad7fa640 10 client.?.objecter ms_handle_connect 0x7f03b00704e0 2024-03-25T18:08:31.005 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.000+0000 7f03ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:31.005 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.000+0000 7f03ad7fa640 10 client.?.objecter ms_handle_connect 0x7f03b00ae6e0 2024-03-25T18:08:31.005 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.000+0000 7f03ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:31.005 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.000+0000 7f03ad7fa640 10 client.?.objecter ms_handle_connect 0x7f03b0142bf0 2024-03-25T18:08:31.005 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.000+0000 7f03ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:31.005 INFO:tasks.ceph.osd.3.smithi156.stderr:==00:00:00:09.182 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:31.006 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.001+0000 7f03ad7fa640 1 -- 172.21.15.156:0/1573043938 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272831581 0 0) 0x7f03b00aaf10 con 0x7f03b0142bf0 2024-03-25T18:08:31.006 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.001+0000 7f03ad7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:08:31.006 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.001+0000 7f03ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:31.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.001+0000 7f03ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:31.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.001+0000 7f03ad7fa640 10 monclient(hunting): my global_id is 4204 2024-03-25T18:08:31.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.001+0000 7f03ad7fa640 1 -- 172.21.15.156:0/1573043938 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0394004eb0 con 0x7f03b0142bf0 2024-03-25T18:08:31.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.002+0000 7f03ad7fa640 1 -- 172.21.15.156:0/1573043938 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3209822993 0 0) 0x7f03b00a9470 con 0x7f03b00ae6e0 2024-03-25T18:08:31.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.002+0000 7f03ad7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:08:31.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.002+0000 7f03ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:31.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.002+0000 7f03ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:31.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.002+0000 7f03ad7fa640 10 monclient(hunting): my global_id is 4206 2024-03-25T18:08:31.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.002+0000 7f03ad7fa640 1 -- 172.21.15.156:0/1573043938 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03b00aaf10 con 0x7f03b00ae6e0 2024-03-25T18:08:31.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.002+0000 7f03ad7fa640 1 -- 172.21.15.156:0/1573043938 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604177752 0 0) 0x7f03b01d73f0 con 0x7f03b00704e0 2024-03-25T18:08:31.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.002+0000 7f03ad7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:08:31.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.002+0000 7f03ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:31.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.002+0000 7f03ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:31.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.002+0000 7f03ad7fa640 10 monclient(hunting): my global_id is 4217 2024-03-25T18:08:31.008 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.002+0000 7f03ad7fa640 1 -- 172.21.15.156:0/1573043938 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03b00a9470 con 0x7f03b00704e0 2024-03-25T18:08:31.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.006+0000 7f03ad7fa640 1 -- 172.21.15.156:0/1573043938 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 236375781 0 0) 0x7f0394004eb0 con 0x7f03b0142bf0 2024-03-25T18:08:31.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.006+0000 7f03ad7fa640 1 -- 172.21.15.156:0/1573043938 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03b01d73f0 con 0x7f03b0142bf0 2024-03-25T18:08:31.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.006+0000 7f03ad7fa640 1 -- 172.21.15.156:0/1573043938 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f03a8001e80 con 0x7f03b0142bf0 2024-03-25T18:08:31.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.006+0000 7f03ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:31.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.006+0000 7f03ad7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:31.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.006+0000 7f03ad7fa640 10 monclient(hunting): dump: 2024-03-25T18:08:31.010 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:08:31.010 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:31.010 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:31.010 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:31.010 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:31.010 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:31.010 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:31.010 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:31.010 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:31.010 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:31.011 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.006+0000 7f03ad7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:31.011 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.007+0000 7f03ad7fa640 1 -- 172.21.15.156:0/1573043938 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1886599183 0 0) 0x7f03b00aaf10 con 0x7f03b00ae6e0 2024-03-25T18:08:31.011 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.007+0000 7f03ad7fa640 1 -- 172.21.15.156:0/1573043938 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0394004eb0 con 0x7f03b00ae6e0 2024-03-25T18:08:31.011 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.007+0000 7f03ad7fa640 1 -- 172.21.15.156:0/1573043938 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f039c001970 con 0x7f03b00ae6e0 2024-03-25T18:08:31.011 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.007+0000 7f03ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:31.011 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.007+0000 7f03ad7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:31.011 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.007+0000 7f03ad7fa640 10 monclient(hunting): dump: 2024-03-25T18:08:31.011 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:08:31.011 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:31.011 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:31.011 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:31.011 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:31.011 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:31.011 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:31.011 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:31.011 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:31.012 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:31.012 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.008+0000 7f03ad7fa640 1 -- 172.21.15.156:0/1573043938 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3948661707 0 0) 0x7f03b00a9470 con 0x7f03b00704e0 2024-03-25T18:08:31.012 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.008+0000 7f03ad7fa640 1 -- 172.21.15.156:0/1573043938 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03b00aaf10 con 0x7f03b00704e0 2024-03-25T18:08:31.012 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.008+0000 7f03ad7fa640 1 -- 172.21.15.156:0/1573043938 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f03a4001740 con 0x7f03b00704e0 2024-03-25T18:08:31.012 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.008+0000 7f03ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:31.012 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.008+0000 7f03ad7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:31.012 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.008+0000 7f03ad7fa640 10 monclient(hunting): dump: 2024-03-25T18:08:31.012 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:08:31.012 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:31.012 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:31.012 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:31.012 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:31.012 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:31.012 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:31.012 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:31.012 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:31.013 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:31.013 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.009+0000 7f03ad7fa640 1 -- 172.21.15.156:0/1573043938 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 42398311 0 0) 0x7f03b01d73f0 con 0x7f03b0142bf0 2024-03-25T18:08:31.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.009+0000 7f03ad7fa640 1 -- 172.21.15.156:0/1573043938 >> v1:172.21.15.156:6790/0 conn(0x7f03b00704e0 legacy=0x7f03b01d6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:31.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.009+0000 7f03ad7fa640 1 -- 172.21.15.156:0/1573043938 >> v1:172.21.15.156:6789/0 conn(0x7f03b00ae6e0 legacy=0x7f03b00a9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:31.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.009+0000 7f03ad7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:08:31.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.009+0000 7f03ad7fa640 1 monclient: found mon.b 2024-03-25T18:08:31.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.009+0000 7f03ad7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:31.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.009+0000 7f03ad7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:31.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.009+0000 7f03ad7fa640 1 -- 172.21.15.156:0/1573043938 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b01d7580 con 0x7f03b0142bf0 2024-03-25T18:08:31.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.009+0000 7f03ad7fa640 10 monclient: _finish_auth 0 2024-03-25T18:08:31.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.009+0000 7f03ad7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:08:31.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.009+0000 7f03ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:31.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.009+0000 7f03b6ffe640 5 monclient: authenticate success, global_id 4204 2024-03-25T18:08:31.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.009+0000 7f03b6ffe640 10 monclient: _renew_subs 2024-03-25T18:08:31.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.009+0000 7f03b6ffe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:31.015 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.009+0000 7f03b6ffe640 1 -- 172.21.15.156:0/1573043938 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03b00aa010 con 0x7f03b0142bf0 2024-03-25T18:08:31.015 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.010+0000 7f03b6ffe640 10 client.4204.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:31.015 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.010+0000 7f03b6ffe640 10 monclient: _renew_subs 2024-03-25T18:08:31.015 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.010+0000 7f03b6ffe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:31.015 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.010+0000 7f03b6ffe640 1 -- 172.21.15.156:0/1573043938 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b0139240 con 0x7f03b0142bf0 2024-03-25T18:08:31.015 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.010+0000 7f03b6ffe640 1 librados: init done 2024-03-25T18:08:31.015 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.011+0000 7f03b6ffe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:31.015 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.011+0000 7f03b6ffe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:31.015 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.011+0000 7f03b6ffe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:31.015 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.011+0000 7f03b6ffe640 1 -- 172.21.15.156:0/1573043938 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03b00ad6c0 con 0x7f03b0142bf0 2024-03-25T18:08:31.025 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.024+0000 7f03ad7fa640 1 -- 172.21.15.156:0/1573043938 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03a8002020 con 0x7f03b0142bf0 2024-03-25T18:08:31.025 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.024+0000 7f03ad7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:31.025 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.024+0000 7f03ad7fa640 1 -- 172.21.15.156:0/1573043938 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f03a8003dc0 con 0x7f03b0142bf0 2024-03-25T18:08:31.025 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.024+0000 7f03ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:31.025 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.024+0000 7f03ad7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:31.025 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.024+0000 7f03ad7fa640 10 monclient: dump: 2024-03-25T18:08:31.025 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:08:31.026 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:31.026 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:31.026 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:31.026 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:31.026 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:31.026 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:31.026 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:31.026 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:31.026 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:31.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.024+0000 7f03aeffd640 4 set_mon_vals no callback set 2024-03-25T18:08:31.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.025+0000 7f03ad7fa640 1 -- 172.21.15.156:0/1573043938 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f03a8019d90 con 0x7f03b0142bf0 2024-03-25T18:08:31.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.026+0000 7f03b4d9a640 10 monclient: get_auth_request con 0x7f039406e060 auth_method 0 2024-03-25T18:08:31.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.026+0000 7f03ad7fa640 10 client.4204.objecter ms_handle_connect 0x7f039406e060 2024-03-25T18:08:31.028 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.027+0000 7f03ad7fa640 1 -- 172.21.15.156:0/1573043938 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f03a8053970 con 0x7f03b0142bf0 2024-03-25T18:08:31.028 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.027+0000 7f03ad7fa640 10 client.4204.objecter ms_dispatch 0x7f03b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:31.028 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.027+0000 7f03ad7fa640 3 client.4204.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:31.028 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.027+0000 7f03ad7fa640 3 client.4204.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:31.028 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.027+0000 7f03ad7fa640 20 client.4204.objecter dump_active .. 0 homeless 2024-03-25T18:08:31.121 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:09.328 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:31.157 INFO:tasks.ceph.osd.1.smithi156.stderr:==00:00:00:09.362 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:31.213 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:09.419 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:31.213 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:09.419 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:31.254 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.254+0000 7f03ad7fa640 1 -- 172.21.15.156:0/1573043938 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f03a804f480 con 0x7f03b0142bf0 2024-03-25T18:08:31.254 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.254+0000 7f03ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:31.255 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.254+0000 7f03ad7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:31.258 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:09.458 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:31.272 INFO:tasks.ceph.osd.2.smithi156.stderr:==00:00:00:09.462 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:31.291 INFO:tasks.ceph.osd.0.smithi156.stderr:==00:00:00:09.483 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:31.303 INFO:tasks.ceph.osd.3.smithi156.stderr:==00:00:00:09.484 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:31.385 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:09.592 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:31.424 INFO:tasks.ceph.osd.1.smithi156.stderr:==00:00:00:09.629 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:31.434 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.434+0000 7f03b6ffe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:31.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.434+0000 7f03b6ffe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:31.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.434+0000 7f03b6ffe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:31.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.434+0000 7f03b6ffe640 1 -- 172.21.15.156:0/1573043938 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03b0074d80 con 0x7f03b0142bf0 2024-03-25T18:08:31.479 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.479+0000 7f03ad7fa640 1 -- 172.21.15.156:0/1573043938 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f03b0074d80 con 0x7f03b0142bf0 2024-03-25T18:08:31.479 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.479+0000 7f03ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:31.480 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.479+0000 7f03ad7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:31.480 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:08:31.480 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:31.482 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.481+0000 7f03b6ffe640 10 librados: watch_flush enter 2024-03-25T18:08:31.482 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.481+0000 7f03b6ffe640 10 librados: watch_flush exit 2024-03-25T18:08:31.482 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.481+0000 7f03b6ffe640 20 client.4204.objecter shutdown clearing up homeless session... 2024-03-25T18:08:31.482 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.481+0000 7f03b6ffe640 10 client.4204.objecter successfully canceled tick 2024-03-25T18:08:31.482 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.481+0000 7f03b6ffe640 1 -- 172.21.15.156:0/1573043938 >> v1:172.21.15.194:6800/2946427713 conn(0x7f039406e060 legacy=0x7f0394070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:31.482 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.481+0000 7f03b6ffe640 10 monclient: shutdown 2024-03-25T18:08:31.482 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.481+0000 7f03b6ffe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:31.482 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.481+0000 7f03b6ffe640 1 -- 172.21.15.156:0/1573043938 >> v1:172.21.15.194:6789/0 conn(0x7f03b0142bf0 legacy=0x7f03b0138b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:31.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.481+0000 7f03b6ffe640 1 -- 172.21.15.156:0/1573043938 shutdown_connections 2024-03-25T18:08:31.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.481+0000 7f03b6ffe640 1 -- 172.21.15.156:0/1573043938 >> 172.21.15.156:0/1573043938 conn(0x7f03b006b630 msgr2=0x7f03b00b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:31.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.481+0000 7f03b6ffe640 1 -- 172.21.15.156:0/1573043938 shutdown_connections 2024-03-25T18:08:31.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.481+0000 7f03b6ffe640 1 -- 172.21.15.156:0/1573043938 wait complete. 2024-03-25T18:08:31.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:31.481+0000 7f03b6ffe640 1 librados: shutdown 2024-03-25T18:08:31.488 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:09.694 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:31.491 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:31.493 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:09.698 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:31.536 INFO:tasks.ceph.osd.2.smithi156.stderr:==00:00:00:09.726 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:31.550 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:09.750 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:31.588 INFO:tasks.ceph.osd.0.smithi156.stderr:==00:00:00:09.780 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:31.596 INFO:tasks.ceph.osd.3.smithi156.stderr:==00:00:00:09.777 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:31.649 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:09.856 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:31.693 INFO:tasks.ceph.osd.1.smithi156.stderr:==00:00:00:09.897 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:31.761 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:09.967 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:31.786 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:09.992 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:31.801 INFO:tasks.ceph.osd.2.smithi156.stderr:==00:00:00:09.991 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:31.838 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:10.038 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:31.876 INFO:tasks.ceph.osd.0.smithi156.stderr:==00:00:00:10.068 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:31.888 INFO:tasks.ceph.osd.3.smithi156.stderr:==00:00:00:10.069 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:31.913 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:10.120 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:31.957 INFO:tasks.ceph.osd.1.smithi156.stderr:==00:00:00:10.162 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:32.025 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:10.231 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:32.073 INFO:tasks.ceph.osd.2.smithi156.stderr:==00:00:00:10.263 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:32.075 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:10.281 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:32.125 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:10.325 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:32.166 INFO:tasks.ceph.osd.0.smithi156.stderr:==00:00:00:10.357 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:32.177 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:10.384 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:32.185 INFO:tasks.ceph.osd.3.smithi156.stderr:==00:00:00:10.366 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:32.224 INFO:tasks.ceph.osd.1.smithi156.stderr:==00:00:00:10.429 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:32.345 INFO:tasks.ceph.osd.2.smithi156.stderr:==00:00:00:10.535 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:32.369 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:10.574 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:32.415 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:10.615 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:32.454 INFO:tasks.ceph.osd.0.smithi156.stderr:==00:00:00:10.646 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:32.488 INFO:tasks.ceph.osd.3.smithi156.stderr:==00:00:00:10.669 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:32.493 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:10.699 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:32.515 INFO:tasks.ceph.osd.1.smithi156.stderr:==00:00:00:10.719 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:32.546 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:10.752 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:32.609 INFO:tasks.ceph.osd.2.smithi156.stderr:==00:00:00:10.799 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:32.644 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:10.851 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:32.697 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:10.904 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:32.742 INFO:tasks.ceph.osd.0.smithi156.stderr:==00:00:00:10.934 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:32.787 INFO:tasks.ceph.osd.1.smithi156.stderr:==00:00:00:10.992 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:32.787 INFO:tasks.ceph.osd.3.smithi156.stderr:==00:00:00:10.968 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:32.903 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:11.108 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:32.957 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:11.163 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:32.962 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:11.162 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:33.016 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:11.216 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:33.031 INFO:tasks.ceph.osd.0.smithi156.stderr:==00:00:00:11.223 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:33.073 INFO:tasks.ceph.osd.2.smithi156.stderr:==00:00:00:11.264 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:33.150 INFO:tasks.ceph.osd.2.smithi156.stderr:==00:00:00:11.340 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:33.271 INFO:tasks.ceph.osd.1.smithi156.stderr:==00:00:00:11.475 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:33.311 INFO:tasks.ceph.osd.3.smithi156.stderr:==00:00:00:11.492 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:33.326 INFO:tasks.ceph.osd.1.smithi156.stderr:==00:00:00:11.531 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:33.366 INFO:tasks.ceph.osd.3.smithi156.stderr:==00:00:00:11.547 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:33.573 INFO:tasks.ceph.osd.0.smithi156.stderr:==00:00:00:11.765 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:33.627 INFO:tasks.ceph.osd.0.smithi156.stderr:==00:00:00:11.819 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:37.936 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:16.142 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:38.148 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:16.355 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:38.297 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:38.390 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.389+0000 7f5cc0281640 10 monclient: get_monmap_and_config 2024-03-25T18:08:38.390 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.389+0000 7f5cc0281640 10 monclient: build_initial_monmap 2024-03-25T18:08:38.390 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.389+0000 7f5cc0281640 10 monclient: monmap: 2024-03-25T18:08:38.391 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:08:38.391 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:38.391 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:08:38.390189+0000 2024-03-25T18:08:38.391 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:08:38.390189+0000 2024-03-25T18:08:38.391 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:38.391 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:38.391 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:38.391 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:08:38.391 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:08:38.391 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:38.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.389+0000 7f5cc0281640 1 Processor -- start 2024-03-25T18:08:38.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.389+0000 7f5cc0281640 1 -- start start 2024-03-25T18:08:38.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.389+0000 7f5cc0281640 10 monclient: init 2024-03-25T18:08:38.392 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.390+0000 7f5cc0281640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:38.393 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.390+0000 7f5cc0281640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:08:38.393 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.390+0000 7f5cc0281640 10 monclient(hunting): picked mon.noname-b con 0x7f5cb80b4d60 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:38.393 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.390+0000 7f5cc0281640 10 monclient(hunting): picked mon.noname-a con 0x7f5cb80b1730 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:38.393 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.390+0000 7f5cc0281640 10 monclient(hunting): picked mon.noname-c con 0x7f5cb814be90 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:38.393 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.390+0000 7f5cc0281640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cb80af940 con 0x7f5cb80b1730 2024-03-25T18:08:38.393 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.390+0000 7f5cc0281640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cb80b0a70 con 0x7f5cb814be90 2024-03-25T18:08:38.393 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.390+0000 7f5cc0281640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cb80b0d50 con 0x7f5cb80b4d60 2024-03-25T18:08:38.393 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.390+0000 7f5cc0281640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:38.393 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.390+0000 7f5cc0281640 10 monclient(hunting): authenticate will time out at 6999.701763s 2024-03-25T18:08:38.393 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.391+0000 7f5cbd81c640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f5cb80b1730 0x7f5cb80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:41658/0 (socket says 172.21.15.156:41658) 2024-03-25T18:08:38.393 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.391+0000 7f5cbd81c640 1 -- 172.21.15.156:0/2486822518 learned_addr learned my addr 172.21.15.156:0/2486822518 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:08:38.395 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.394+0000 7f5cbd01b640 1 -- 172.21.15.156:0/2486822518 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300342806 0 0) 0x7f5cb80af940 con 0x7f5cb80b1730 2024-03-25T18:08:38.395 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.394+0000 7f5cbd01b640 10 monclient(hunting): _negotiate 2024-03-25T18:08:38.395 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.394+0000 7f5cbd01b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:38.395 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.394+0000 7f5cbd01b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:38.395 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.394+0000 7f5cbd01b640 10 monclient(hunting): my global_id is 4209 2024-03-25T18:08:38.395 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.394+0000 7f5cbd01b640 1 -- 172.21.15.156:0/2486822518 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ca4003520 con 0x7f5cb80b1730 2024-03-25T18:08:38.396 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.395+0000 7f5cbd01b640 1 -- 172.21.15.156:0/2486822518 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1086928140 0 0) 0x7f5cb80b0a70 con 0x7f5cb814be90 2024-03-25T18:08:38.396 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.395+0000 7f5cbd01b640 10 monclient(hunting): _negotiate 2024-03-25T18:08:38.396 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.395+0000 7f5cbd01b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:38.396 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.395+0000 7f5cbd01b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:38.396 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.395+0000 7f5cbd01b640 10 monclient(hunting): my global_id is 4207 2024-03-25T18:08:38.396 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.395+0000 7f5cbd01b640 1 -- 172.21.15.156:0/2486822518 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cb80af940 con 0x7f5cb814be90 2024-03-25T18:08:38.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.398+0000 7f5cbd01b640 1 -- 172.21.15.156:0/2486822518 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1283256028 0 0) 0x7f5ca4003520 con 0x7f5cb80b1730 2024-03-25T18:08:38.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.398+0000 7f5cbd01b640 1 -- 172.21.15.156:0/2486822518 >> v1:172.21.15.156:6790/0 conn(0x7f5cb80b4d60 legacy=0x7f5cb813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:08:38.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.398+0000 7f5cbd01b640 1 -- 172.21.15.156:0/2486822518 >> v1:172.21.15.194:6789/0 conn(0x7f5cb814be90 legacy=0x7f5cb814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:38.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.398+0000 7f5cbd01b640 10 monclient: _finish_hunting 0 2024-03-25T18:08:38.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.398+0000 7f5cbd01b640 1 monclient: found mon.noname-a 2024-03-25T18:08:38.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.398+0000 7f5cbd01b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:38.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.398+0000 7f5cbd01b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:08:38.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.398+0000 7f5cbd01b640 1 -- 172.21.15.156:0/2486822518 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb80b1030 con 0x7f5cb80b1730 2024-03-25T18:08:38.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.398+0000 7f5cbd01b640 10 monclient: _finish_auth 0 2024-03-25T18:08:38.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.398+0000 7f5cbd01b640 10 monclient: _check_auth_tickets 2024-03-25T18:08:38.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.398+0000 7f5cbd01b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:38.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.398+0000 7f5cbd01b640 1 -- 172.21.15.156:0/2486822518 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5ca8001690 con 0x7f5cb80b1730 2024-03-25T18:08:38.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.398+0000 7f5cbd01b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:38.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.398+0000 7f5cbd01b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:08:38.401 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.398+0000 7f5cbd01b640 10 monclient: dump: 2024-03-25T18:08:38.401 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:08:38.401 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:38.401 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:38.401 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:38.401 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:38.401 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:38.401 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:38.401 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:38.401 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:38.401 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:38.401 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.398+0000 7f5cc0281640 5 monclient: authenticate success, global_id 4209 2024-03-25T18:08:38.401 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.398+0000 7f5cc0281640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:38.405 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.399+0000 7f5cbd01b640 1 -- 172.21.15.156:0/2486822518 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ca80017f0 con 0x7f5cb80b1730 2024-03-25T18:08:38.405 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.399+0000 7f5cbd01b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:38.405 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.399+0000 7f5cbd01b640 1 -- 172.21.15.156:0/2486822518 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5ca8001c00 con 0x7f5cb80b1730 2024-03-25T18:08:38.405 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.399+0000 7f5cbd01b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:38.405 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.399+0000 7f5cbd01b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:38.405 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.399+0000 7f5cbd01b640 10 monclient: dump: 2024-03-25T18:08:38.405 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:08:38.405 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:38.406 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:38.406 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:38.406 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:38.406 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:38.406 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:38.406 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:38.406 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:38.406 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:38.406 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.399+0000 7f5cc0281640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:38.406 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.399+0000 7f5cc0281640 4 set_mon_vals no callback set 2024-03-25T18:08:38.406 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.400+0000 7f5cc0281640 10 monclient: shutdown 2024-03-25T18:08:38.406 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.400+0000 7f5cc0281640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:38.406 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.400+0000 7f5cc0281640 1 -- 172.21.15.156:0/2486822518 >> v1:172.21.15.156:6789/0 conn(0x7f5cb80b1730 legacy=0x7f5cb80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:38.406 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.401+0000 7f5cc0281640 1 -- 172.21.15.156:0/2486822518 shutdown_connections 2024-03-25T18:08:38.406 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.401+0000 7f5cc0281640 1 -- 172.21.15.156:0/2486822518 >> 172.21.15.156:0/2486822518 conn(0x7f5cb80617a0 msgr2=0x7f5cb8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:38.406 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.401+0000 7f5cc0281640 1 -- 172.21.15.156:0/2486822518 shutdown_connections 2024-03-25T18:08:38.407 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.401+0000 7f5cc0281640 1 -- 172.21.15.156:0/2486822518 wait complete. 2024-03-25T18:08:38.407 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.402+0000 7f5cc0281640 10 monclient: build_initial_monmap 2024-03-25T18:08:38.407 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.402+0000 7f5cc0281640 10 monclient: monmap: 2024-03-25T18:08:38.407 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:08:38.407 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:38.407 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:08:38.407 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:08:38.407 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:38.407 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:08:38.407 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:38.407 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:08:38.407 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:08:38.407 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:38.407 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.402+0000 7f5cc0281640 1 librados: starting msgr at 2024-03-25T18:08:38.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.402+0000 7f5cc0281640 1 librados: starting objecter 2024-03-25T18:08:38.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.402+0000 7f5cc0281640 1 Processor -- start 2024-03-25T18:08:38.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.402+0000 7f5cc0281640 1 -- start start 2024-03-25T18:08:38.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.402+0000 7f5cc0281640 1 librados: setting wanted keys 2024-03-25T18:08:38.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.402+0000 7f5cc0281640 1 librados: calling monclient init 2024-03-25T18:08:38.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.402+0000 7f5cc0281640 10 monclient: init 2024-03-25T18:08:38.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.403+0000 7f5cc0281640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:38.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.403+0000 7f5cc0281640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:08:38.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.403+0000 7f5cc0281640 10 monclient(hunting): picked mon.noname-b con 0x7f5cb80b1730 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:38.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.403+0000 7f5cc0281640 10 monclient(hunting): picked mon.noname-a con 0x7f5cb80b4d60 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:38.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.403+0000 7f5cc0281640 10 monclient(hunting): picked mon.noname-c con 0x7f5cb814be90 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:38.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.403+0000 7f5cc0281640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cb81379d0 con 0x7f5cb80b4d60 2024-03-25T18:08:38.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.403+0000 7f5cc0281640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cb81444a0 con 0x7f5cb80b1730 2024-03-25T18:08:38.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.403+0000 7f5cc0281640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cb8137390 con 0x7f5cb814be90 2024-03-25T18:08:38.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.403+0000 7f5cc0281640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:38.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.403+0000 7f5cc0281640 10 monclient(hunting): authenticate will time out at 6999.714446s 2024-03-25T18:08:38.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.403+0000 7f5cbe01d640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f5cb80b1730 0x7f5cb812fd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:35498/0 (socket says 172.21.15.156:35498) 2024-03-25T18:08:38.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.404+0000 7f5cbe01d640 1 -- 172.21.15.156:0/631848512 learned_addr learned my addr 172.21.15.156:0/631848512 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:08:38.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.404+0000 7f5caeffd640 10 client.?.objecter ms_handle_connect 0x7f5cb80b4d60 2024-03-25T18:08:38.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.404+0000 7f5caeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:38.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.404+0000 7f5caeffd640 10 client.?.objecter ms_handle_connect 0x7f5cb80b1730 2024-03-25T18:08:38.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.404+0000 7f5caeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:38.410 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.406+0000 7f5caeffd640 1 -- 172.21.15.156:0/631848512 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464113312 0 0) 0x7f5cb81379d0 con 0x7f5cb80b4d60 2024-03-25T18:08:38.410 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.406+0000 7f5caeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:38.410 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.406+0000 7f5caeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:38.410 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.406+0000 7f5caeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:38.410 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.406+0000 7f5caeffd640 10 monclient(hunting): my global_id is 4212 2024-03-25T18:08:38.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.406+0000 7f5caeffd640 1 -- 172.21.15.156:0/631848512 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c9c004eb0 con 0x7f5cb80b4d60 2024-03-25T18:08:38.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.407+0000 7f5caeffd640 1 -- 172.21.15.156:0/631848512 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3043165131 0 0) 0x7f5cb81444a0 con 0x7f5cb80b1730 2024-03-25T18:08:38.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.407+0000 7f5caeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:38.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.407+0000 7f5caeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:38.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.407+0000 7f5caeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:38.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.407+0000 7f5caeffd640 10 monclient(hunting): my global_id is 4210 2024-03-25T18:08:38.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.407+0000 7f5caeffd640 1 -- 172.21.15.156:0/631848512 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cb81379d0 con 0x7f5cb80b1730 2024-03-25T18:08:38.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.411+0000 7f5caeffd640 1 -- 172.21.15.156:0/631848512 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2128403027 0 0) 0x7f5c9c004eb0 con 0x7f5cb80b4d60 2024-03-25T18:08:38.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.411+0000 7f5caeffd640 1 -- 172.21.15.156:0/631848512 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5cb81444a0 con 0x7f5cb80b4d60 2024-03-25T18:08:38.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.411+0000 7f5caeffd640 1 -- 172.21.15.156:0/631848512 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5ca8001bc0 con 0x7f5cb80b4d60 2024-03-25T18:08:38.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.411+0000 7f5caeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:38.412 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.411+0000 7f5caeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:08:38.412 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.411+0000 7f5caeffd640 10 monclient(hunting): dump: 2024-03-25T18:08:38.412 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:08:38.412 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:38.412 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:38.412 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:38.412 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:38.412 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:38.412 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:38.412 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:38.412 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:38.412 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:38.412 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.411+0000 7f5caeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:38.414 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.413+0000 7f5caeffd640 10 client.?.objecter ms_handle_connect 0x7f5cb814be90 2024-03-25T18:08:38.414 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.413+0000 7f5caeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:38.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.413+0000 7f5caeffd640 1 -- 172.21.15.156:0/631848512 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4095970420 0 0) 0x7f5cb81379d0 con 0x7f5cb80b1730 2024-03-25T18:08:38.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.413+0000 7f5caeffd640 1 -- 172.21.15.156:0/631848512 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c9c004eb0 con 0x7f5cb80b1730 2024-03-25T18:08:38.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.413+0000 7f5caeffd640 1 -- 172.21.15.156:0/631848512 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5cb4001920 con 0x7f5cb80b1730 2024-03-25T18:08:38.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.413+0000 7f5caeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:38.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.413+0000 7f5caeffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:38.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.413+0000 7f5caeffd640 10 monclient(hunting): dump: 2024-03-25T18:08:38.415 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:08:38.415 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:38.415 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:38.415 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:38.415 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:38.415 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:38.415 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:38.415 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:38.416 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:38.416 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:38.417 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.417+0000 7f5caeffd640 1 -- 172.21.15.156:0/631848512 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2382236536 0 0) 0x7f5cb8137390 con 0x7f5cb814be90 2024-03-25T18:08:38.418 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.417+0000 7f5caeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:38.418 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.417+0000 7f5caeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:38.418 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.417+0000 7f5caeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:38.418 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.417+0000 7f5caeffd640 10 monclient(hunting): my global_id is 4220 2024-03-25T18:08:38.418 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.417+0000 7f5caeffd640 1 -- 172.21.15.156:0/631848512 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cb81379d0 con 0x7f5cb814be90 2024-03-25T18:08:38.421 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.418+0000 7f5caeffd640 1 -- 172.21.15.156:0/631848512 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1683056303 0 0) 0x7f5c9c004eb0 con 0x7f5cb80b1730 2024-03-25T18:08:38.421 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.418+0000 7f5caeffd640 1 -- 172.21.15.156:0/631848512 >> v1:172.21.15.156:6790/0 conn(0x7f5cb814be90 legacy=0x7f5cb81db310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:38.421 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.418+0000 7f5caeffd640 1 -- 172.21.15.156:0/631848512 >> v1:172.21.15.156:6789/0 conn(0x7f5cb80b4d60 legacy=0x7f5cb81dac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:38.421 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.418+0000 7f5caeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:08:38.421 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.418+0000 7f5caeffd640 1 monclient: found mon.b 2024-03-25T18:08:38.421 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.418+0000 7f5caeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:38.421 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.418+0000 7f5caeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:38.421 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.418+0000 7f5caeffd640 1 -- 172.21.15.156:0/631848512 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb8137520 con 0x7f5cb80b1730 2024-03-25T18:08:38.421 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.418+0000 7f5caeffd640 10 monclient: _finish_auth 0 2024-03-25T18:08:38.422 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.418+0000 7f5caeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:08:38.422 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.418+0000 7f5caeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:38.422 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.418+0000 7f5cc0281640 5 monclient: authenticate success, global_id 4210 2024-03-25T18:08:38.422 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.419+0000 7f5cc0281640 10 monclient: _renew_subs 2024-03-25T18:08:38.422 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.419+0000 7f5cc0281640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:38.422 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.419+0000 7f5cc0281640 1 -- 172.21.15.156:0/631848512 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cb8143e30 con 0x7f5cb80b1730 2024-03-25T18:08:38.422 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.419+0000 7f5cc0281640 10 client.4210.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:38.422 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.419+0000 7f5cc0281640 10 monclient: _renew_subs 2024-03-25T18:08:38.422 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.419+0000 7f5cc0281640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:38.422 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.419+0000 7f5cc0281640 1 -- 172.21.15.156:0/631848512 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cb8130670 con 0x7f5cb80b1730 2024-03-25T18:08:38.422 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.419+0000 7f5cc0281640 1 librados: init done 2024-03-25T18:08:38.422 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.420+0000 7f5cacff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:38.422 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.420+0000 7f5cacff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:38.422 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.420+0000 7f5cacff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:38.423 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.420+0000 7f5cacff9640 1 -- 172.21.15.156:0/631848512 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ca0005080 con 0x7f5cb80b1730 2024-03-25T18:08:38.424 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:16.629 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:38.557 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:16.757 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:38.634 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.634+0000 7f5caeffd640 1 -- 172.21.15.156:0/631848512 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5cb40035f0 con 0x7f5cb80b1730 2024-03-25T18:08:38.634 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.634+0000 7f5caeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:38.634 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.634+0000 7f5caeffd640 1 -- 172.21.15.156:0/631848512 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5cb40039c0 con 0x7f5cb80b1730 2024-03-25T18:08:38.634 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.634+0000 7f5caeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:38.635 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.634+0000 7f5caeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:38.635 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.634+0000 7f5cbc81a640 4 set_mon_vals no callback set 2024-03-25T18:08:38.635 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.634+0000 7f5caeffd640 10 monclient: dump: 2024-03-25T18:08:38.635 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:08:38.635 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:38.635 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:38.635 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:38.635 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:38.635 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:38.635 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:38.635 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:38.635 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:38.635 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:38.756 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.755+0000 7f5caeffd640 1 -- 172.21.15.156:0/631848512 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f5cb4003b20 con 0x7f5cb80b1730 2024-03-25T18:08:38.757 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.756+0000 7f5cbd81c640 10 monclient: get_auth_request con 0x7f5c9c06df90 auth_method 0 2024-03-25T18:08:38.757 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.756+0000 7f5caeffd640 10 client.4210.objecter ms_handle_connect 0x7f5c9c06df90 2024-03-25T18:08:38.758 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.758+0000 7f5caeffd640 1 -- 172.21.15.156:0/631848512 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f5cb40547d0 con 0x7f5cb80b1730 2024-03-25T18:08:38.759 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.758+0000 7f5caeffd640 10 client.4210.objecter ms_dispatch 0x7f5cb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:38.759 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.758+0000 7f5caeffd640 3 client.4210.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:38.759 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.758+0000 7f5caeffd640 3 client.4210.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:38.759 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.758+0000 7f5caeffd640 20 client.4210.objecter dump_active .. 0 homeless 2024-03-25T18:08:38.760 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.760+0000 7f5caeffd640 1 -- 172.21.15.156:0/631848512 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5cb40502e0 con 0x7f5cb80b1730 2024-03-25T18:08:38.760 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.760+0000 7f5caeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:38.761 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:38.760+0000 7f5caeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:38.918 INFO:tasks.ceph.osd.2.smithi156.stderr:==00:00:00:17.108 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:39.100 INFO:tasks.ceph.osd.1.smithi156.stderr:==00:00:00:17.304 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:39.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:39.131+0000 7f5cacff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:39.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:39.131+0000 7f5cacff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:39.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:39.131+0000 7f5cacff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:39.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:39.131+0000 7f5cacff9640 1 -- 172.21.15.156:0/631848512 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ca0005330 con 0x7f5cb80b1730 2024-03-25T18:08:39.147 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:39.146+0000 7f5caeffd640 1 -- 172.21.15.156:0/631848512 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f5cb4052210 con 0x7f5cb80b1730 2024-03-25T18:08:39.147 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:39.146+0000 7f5caeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:39.147 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:39.146+0000 7f5caeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:39.147 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:08:39.147 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:39.149 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:39.148+0000 7f5cacff9640 10 librados: watch_flush enter 2024-03-25T18:08:39.149 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:39.148+0000 7f5cacff9640 10 librados: watch_flush exit 2024-03-25T18:08:39.150 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:39.148+0000 7f5cacff9640 20 client.4210.objecter shutdown clearing up homeless session... 2024-03-25T18:08:39.150 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:39.148+0000 7f5cacff9640 10 client.4210.objecter successfully canceled tick 2024-03-25T18:08:39.150 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:39.148+0000 7f5cacff9640 1 -- 172.21.15.156:0/631848512 >> v1:172.21.15.194:6800/2946427713 conn(0x7f5c9c06df90 legacy=0x7f5c9c070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:39.150 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:39.148+0000 7f5cacff9640 10 monclient: shutdown 2024-03-25T18:08:39.150 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:39.149+0000 7f5cacff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:39.150 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:39.149+0000 7f5cacff9640 1 -- 172.21.15.156:0/631848512 >> v1:172.21.15.194:6789/0 conn(0x7f5cb80b1730 legacy=0x7f5cb812fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:39.150 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:39.149+0000 7f5cacff9640 1 -- 172.21.15.156:0/631848512 shutdown_connections 2024-03-25T18:08:39.150 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:39.149+0000 7f5cacff9640 1 -- 172.21.15.156:0/631848512 >> 172.21.15.156:0/631848512 conn(0x7f5cb80617a0 msgr2=0x7f5cb8137f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:39.150 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:39.149+0000 7f5cacff9640 1 -- 172.21.15.156:0/631848512 shutdown_connections 2024-03-25T18:08:39.150 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:39.149+0000 7f5cacff9640 1 -- 172.21.15.156:0/631848512 wait complete. 2024-03-25T18:08:39.150 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:39.149+0000 7f5cacff9640 1 librados: shutdown 2024-03-25T18:08:39.164 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:39.402 INFO:tasks.ceph.osd.3.smithi156.stderr:==00:00:00:17.583 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:39.666 INFO:tasks.ceph.osd.0.smithi156.stderr:==00:00:00:17.857 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:43.694 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:21.900 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:43.700 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:21.906 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:43.851 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:22.058 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:43.854 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:22.062 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:43.977 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:22.182 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:43.981 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:22.186 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:44.725 INFO:tasks.ceph.osd.1.smithi156.stderr:==00:00:00:22.929 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:44.729 INFO:tasks.ceph.osd.1.smithi156.stderr:==00:00:00:22.933 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:44.858 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:23.058 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:44.861 INFO:tasks.ceph.osd.2.smithi156.stderr:==00:00:00:23.051 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:44.864 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:23.064 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:44.865 INFO:tasks.ceph.osd.2.smithi156.stderr:==00:00:00:23.055 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:45.210 INFO:tasks.ceph.osd.3.smithi156.stderr:==00:00:00:23.391 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:45.213 INFO:tasks.ceph.osd.3.smithi156.stderr:==00:00:00:23.395 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:45.483 INFO:tasks.ceph.osd.0.smithi156.stderr:==00:00:00:23.674 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:45.489 INFO:tasks.ceph.osd.0.smithi156.stderr:==00:00:00:23.680 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:45.970 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:46.090 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.089+0000 7f8c13482640 10 monclient: get_monmap_and_config 2024-03-25T18:08:46.090 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.089+0000 7f8c13482640 10 monclient: build_initial_monmap 2024-03-25T18:08:46.090 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.089+0000 7f8c13482640 10 monclient: monmap: 2024-03-25T18:08:46.090 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:08:46.090 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:46.090 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:08:46.090630+0000 2024-03-25T18:08:46.090 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:08:46.090630+0000 2024-03-25T18:08:46.090 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:46.090 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:46.091 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:46.091 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:08:46.091 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:08:46.091 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:46.091 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.090+0000 7f8c13482640 1 Processor -- start 2024-03-25T18:08:46.091 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.090+0000 7f8c13482640 1 -- start start 2024-03-25T18:08:46.091 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.090+0000 7f8c13482640 10 monclient: init 2024-03-25T18:08:46.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.091+0000 7f8c13482640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:46.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.091+0000 7f8c13482640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:08:46.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.091+0000 7f8c13482640 10 monclient(hunting): picked mon.noname-c con 0x7f8c0c0b4d60 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:46.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.091+0000 7f8c13482640 10 monclient(hunting): picked mon.noname-a con 0x7f8c0c0b1730 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:46.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.091+0000 7f8c13482640 10 monclient(hunting): picked mon.noname-b con 0x7f8c0c14be90 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:46.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.091+0000 7f8c13482640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c0c0af940 con 0x7f8c0c0b1730 2024-03-25T18:08:46.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.091+0000 7f8c13482640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c0c0b0a70 con 0x7f8c0c0b4d60 2024-03-25T18:08:46.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.091+0000 7f8c13482640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c0c0b0d50 con 0x7f8c0c14be90 2024-03-25T18:08:46.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.091+0000 7f8c13482640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:46.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.091+0000 7f8c13482640 10 monclient(hunting): authenticate will time out at 7007.402699s 2024-03-25T18:08:46.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.092+0000 7f8c1121e640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f8c0c0b4d60 0x7f8c0c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:43764/0 (socket says 172.21.15.156:43764) 2024-03-25T18:08:46.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.092+0000 7f8c1121e640 1 -- 172.21.15.156:0/326858697 learned_addr learned my addr 172.21.15.156:0/326858697 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:08:46.095 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.094+0000 7f8c03fff640 1 -- 172.21.15.156:0/326858697 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535822921 0 0) 0x7f8c0c0b0a70 con 0x7f8c0c0b4d60 2024-03-25T18:08:46.095 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.094+0000 7f8c03fff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:46.095 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.094+0000 7f8c03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:46.095 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.094+0000 7f8c03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:46.095 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.094+0000 7f8c03fff640 10 monclient(hunting): my global_id is 4213 2024-03-25T18:08:46.095 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.094+0000 7f8c03fff640 1 -- 172.21.15.156:0/326858697 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bf8003520 con 0x7f8c0c0b4d60 2024-03-25T18:08:46.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.095+0000 7f8c03fff640 1 -- 172.21.15.156:0/326858697 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479055886 0 0) 0x7f8c0c0b0d50 con 0x7f8c0c14be90 2024-03-25T18:08:46.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.095+0000 7f8c03fff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:46.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.095+0000 7f8c03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:46.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.095+0000 7f8c03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:46.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.095+0000 7f8c03fff640 10 monclient(hunting): my global_id is 4223 2024-03-25T18:08:46.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.095+0000 7f8c03fff640 1 -- 172.21.15.156:0/326858697 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c0c0b0a70 con 0x7f8c0c14be90 2024-03-25T18:08:46.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.096+0000 7f8c03fff640 1 -- 172.21.15.156:0/326858697 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1455102817 0 0) 0x7f8c0c0af940 con 0x7f8c0c0b1730 2024-03-25T18:08:46.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.096+0000 7f8c03fff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:46.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.096+0000 7f8c03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:46.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.096+0000 7f8c03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:46.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.096+0000 7f8c03fff640 10 monclient(hunting): my global_id is 4215 2024-03-25T18:08:46.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.096+0000 7f8c03fff640 1 -- 172.21.15.156:0/326858697 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c0c0b0d50 con 0x7f8c0c0b1730 2024-03-25T18:08:46.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.097+0000 7f8c03fff640 1 -- 172.21.15.156:0/326858697 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3246198698 0 0) 0x7f8bf8003520 con 0x7f8c0c0b4d60 2024-03-25T18:08:46.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.097+0000 7f8c03fff640 1 -- 172.21.15.156:0/326858697 >> v1:172.21.15.156:6790/0 conn(0x7f8c0c14be90 legacy=0x7f8c0c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.097+0000 7f8c03fff640 1 -- 172.21.15.156:0/326858697 >> v1:172.21.15.156:6789/0 conn(0x7f8c0c0b1730 legacy=0x7f8c0c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.097+0000 7f8c03fff640 10 monclient: _finish_hunting 0 2024-03-25T18:08:46.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.097+0000 7f8c03fff640 1 monclient: found mon.noname-c 2024-03-25T18:08:46.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.097+0000 7f8c03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:46.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.097+0000 7f8c03fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:08:46.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.097+0000 7f8c03fff640 1 -- 172.21.15.156:0/326858697 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c0c0b1030 con 0x7f8c0c0b4d60 2024-03-25T18:08:46.099 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.097+0000 7f8c03fff640 10 monclient: _finish_auth 0 2024-03-25T18:08:46.099 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.097+0000 7f8c03fff640 10 monclient: _check_auth_tickets 2024-03-25T18:08:46.099 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.097+0000 7f8c03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:46.099 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.097+0000 7f8c03fff640 1 -- 172.21.15.156:0/326858697 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f8c08001690 con 0x7f8c0c0b4d60 2024-03-25T18:08:46.099 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.097+0000 7f8c03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:46.099 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.097+0000 7f8c03fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:46.099 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.097+0000 7f8c03fff640 10 monclient: dump: 2024-03-25T18:08:46.099 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:08:46.099 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:46.099 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:46.099 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:46.099 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:46.099 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:46.100 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:46.100 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:46.100 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:46.100 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:46.100 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.097+0000 7f8c13482640 5 monclient: authenticate success, global_id 4213 2024-03-25T18:08:46.100 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.097+0000 7f8c13482640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:46.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.099+0000 7f8c03fff640 1 -- 172.21.15.156:0/326858697 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c080017f0 con 0x7f8c0c0b4d60 2024-03-25T18:08:46.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.099+0000 7f8c03fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:46.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.099+0000 7f8c03fff640 1 -- 172.21.15.156:0/326858697 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f8c08001c00 con 0x7f8c0c0b4d60 2024-03-25T18:08:46.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.099+0000 7f8c03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:46.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.099+0000 7f8c03fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:46.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.099+0000 7f8c03fff640 10 monclient: dump: 2024-03-25T18:08:46.101 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:08:46.101 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:46.101 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:46.101 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:46.101 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:46.101 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:46.101 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:46.101 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:46.101 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:46.102 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:46.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.099+0000 7f8c13482640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:46.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.099+0000 7f8c13482640 4 set_mon_vals no callback set 2024-03-25T18:08:46.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.100+0000 7f8c13482640 10 monclient: shutdown 2024-03-25T18:08:46.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.100+0000 7f8c13482640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:46.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.100+0000 7f8c13482640 1 -- 172.21.15.156:0/326858697 >> v1:172.21.15.194:6789/0 conn(0x7f8c0c0b4d60 legacy=0x7f8c0c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.100+0000 7f8c13482640 1 -- 172.21.15.156:0/326858697 shutdown_connections 2024-03-25T18:08:46.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.100+0000 7f8c13482640 1 -- 172.21.15.156:0/326858697 >> 172.21.15.156:0/326858697 conn(0x7f8c0c0617a0 msgr2=0x7f8c0c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:46.104 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.101+0000 7f8c13482640 1 -- 172.21.15.156:0/326858697 shutdown_connections 2024-03-25T18:08:46.104 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.101+0000 7f8c13482640 1 -- 172.21.15.156:0/326858697 wait complete. 2024-03-25T18:08:46.105 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.101+0000 7f8c13482640 10 monclient: build_initial_monmap 2024-03-25T18:08:46.105 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.101+0000 7f8c13482640 10 monclient: monmap: 2024-03-25T18:08:46.105 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:08:46.105 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:46.105 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:08:46.105 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:08:46.105 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:46.105 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:08:46.105 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:46.105 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:08:46.105 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:08:46.105 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:46.105 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.102+0000 7f8c13482640 1 librados: starting msgr at 2024-03-25T18:08:46.105 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.102+0000 7f8c13482640 1 librados: starting objecter 2024-03-25T18:08:46.105 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.102+0000 7f8c13482640 1 Processor -- start 2024-03-25T18:08:46.106 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.102+0000 7f8c13482640 1 -- start start 2024-03-25T18:08:46.106 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.102+0000 7f8c13482640 1 librados: setting wanted keys 2024-03-25T18:08:46.106 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.102+0000 7f8c13482640 1 librados: calling monclient init 2024-03-25T18:08:46.106 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.102+0000 7f8c13482640 10 monclient: init 2024-03-25T18:08:46.106 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.103+0000 7f8c13482640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:46.106 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.103+0000 7f8c13482640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:08:46.106 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.103+0000 7f8c13482640 10 monclient(hunting): picked mon.noname-a con 0x7f8c0c0b1730 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:46.106 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.103+0000 7f8c13482640 10 monclient(hunting): picked mon.noname-b con 0x7f8c0c0b4d60 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:46.106 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.103+0000 7f8c13482640 10 monclient(hunting): picked mon.noname-c con 0x7f8c0c14be90 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:46.106 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.103+0000 7f8c13482640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c0c141f10 con 0x7f8c0c0b1730 2024-03-25T18:08:46.106 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.103+0000 7f8c13482640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c0c1d7ab0 con 0x7f8c0c0b4d60 2024-03-25T18:08:46.106 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.103+0000 7f8c13482640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c0c1d86e0 con 0x7f8c0c14be90 2024-03-25T18:08:46.106 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.103+0000 7f8c13482640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:46.106 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.103+0000 7f8c13482640 10 monclient(hunting): authenticate will time out at 7007.414462s 2024-03-25T18:08:46.106 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.103+0000 7f8c11a1f640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f8c0c14be90 0x7f8c0c1e31e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:49710/0 (socket says 172.21.15.156:49710) 2024-03-25T18:08:46.107 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.103+0000 7f8c11a1f640 1 -- 172.21.15.156:0/651392821 learned_addr learned my addr 172.21.15.156:0/651392821 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:08:46.107 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.104+0000 7f8c01ffb640 10 client.?.objecter ms_handle_connect 0x7f8c0c14be90 2024-03-25T18:08:46.107 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.104+0000 7f8c01ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:46.108 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.104+0000 7f8c01ffb640 10 client.?.objecter ms_handle_connect 0x7f8c0c0b1730 2024-03-25T18:08:46.108 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.104+0000 7f8c01ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:46.108 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.105+0000 7f8c01ffb640 10 client.?.objecter ms_handle_connect 0x7f8c0c0b4d60 2024-03-25T18:08:46.108 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.105+0000 7f8c01ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:46.108 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.107+0000 7f8c01ffb640 1 -- 172.21.15.156:0/651392821 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 90837351 0 0) 0x7f8c0c1d86e0 con 0x7f8c0c14be90 2024-03-25T18:08:46.108 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.107+0000 7f8c01ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:08:46.108 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.107+0000 7f8c01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:46.108 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.107+0000 7f8c01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:46.108 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.107+0000 7f8c01ffb640 10 monclient(hunting): my global_id is 4226 2024-03-25T18:08:46.109 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.107+0000 7f8c01ffb640 1 -- 172.21.15.156:0/651392821 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bf0004eb0 con 0x7f8c0c14be90 2024-03-25T18:08:46.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.108+0000 7f8c01ffb640 1 -- 172.21.15.156:0/651392821 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2875411391 0 0) 0x7f8c0c1d7ab0 con 0x7f8c0c0b4d60 2024-03-25T18:08:46.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.108+0000 7f8c01ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:08:46.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.108+0000 7f8c01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:46.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.108+0000 7f8c01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:46.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.108+0000 7f8c01ffb640 10 monclient(hunting): my global_id is 4216 2024-03-25T18:08:46.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.108+0000 7f8c01ffb640 1 -- 172.21.15.156:0/651392821 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c0c1d86e0 con 0x7f8c0c0b4d60 2024-03-25T18:08:46.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.108+0000 7f8c01ffb640 1 -- 172.21.15.156:0/651392821 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1278649283 0 0) 0x7f8c0c141f10 con 0x7f8c0c0b1730 2024-03-25T18:08:46.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.108+0000 7f8c01ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:08:46.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.108+0000 7f8c01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:46.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.108+0000 7f8c01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:46.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.108+0000 7f8c01ffb640 10 monclient(hunting): my global_id is 4218 2024-03-25T18:08:46.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.108+0000 7f8c01ffb640 1 -- 172.21.15.156:0/651392821 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c0c1d7ab0 con 0x7f8c0c0b1730 2024-03-25T18:08:46.114 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.113+0000 7f8c01ffb640 1 -- 172.21.15.156:0/651392821 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3582461771 0 0) 0x7f8c0c1d86e0 con 0x7f8c0c0b4d60 2024-03-25T18:08:46.114 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.113+0000 7f8c01ffb640 1 -- 172.21.15.156:0/651392821 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c0c141f10 con 0x7f8c0c0b4d60 2024-03-25T18:08:46.114 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.113+0000 7f8c01ffb640 1 -- 172.21.15.156:0/651392821 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f8bfc001870 con 0x7f8c0c0b4d60 2024-03-25T18:08:46.114 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.113+0000 7f8c01ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:46.114 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.113+0000 7f8c01ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:46.114 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.113+0000 7f8c01ffb640 10 monclient(hunting): dump: 2024-03-25T18:08:46.114 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:08:46.114 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:46.114 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:46.114 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:46.114 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:46.114 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:46.115 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:46.115 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:46.115 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:46.115 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:46.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.113+0000 7f8c01ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:46.116 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.116+0000 7f8c01ffb640 1 -- 172.21.15.156:0/651392821 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1725290613 0 0) 0x7f8bf0004eb0 con 0x7f8c0c14be90 2024-03-25T18:08:46.116 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.116+0000 7f8c01ffb640 1 -- 172.21.15.156:0/651392821 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c0c1d86e0 con 0x7f8c0c14be90 2024-03-25T18:08:46.119 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.117+0000 7f8c01ffb640 1 -- 172.21.15.156:0/651392821 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1658937520 0 0) 0x7f8c0c1d7ab0 con 0x7f8c0c0b1730 2024-03-25T18:08:46.119 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.117+0000 7f8c01ffb640 1 -- 172.21.15.156:0/651392821 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8bf0004eb0 con 0x7f8c0c0b1730 2024-03-25T18:08:46.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.117+0000 7f8c01ffb640 1 -- 172.21.15.156:0/651392821 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f8c08001b00 con 0x7f8c0c0b1730 2024-03-25T18:08:46.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.117+0000 7f8c01ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:46.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.117+0000 7f8c01ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:46.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.117+0000 7f8c01ffb640 10 monclient(hunting): dump: 2024-03-25T18:08:46.120 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:08:46.120 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:46.120 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:46.120 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:46.120 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:46.120 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:46.120 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:46.120 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:46.120 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:46.120 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:46.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.117+0000 7f8c01ffb640 1 -- 172.21.15.156:0/651392821 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3121579278 0 0) 0x7f8c0c141f10 con 0x7f8c0c0b4d60 2024-03-25T18:08:46.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.117+0000 7f8c01ffb640 1 -- 172.21.15.156:0/651392821 >> v1:172.21.15.156:6790/0 conn(0x7f8c0c14be90 legacy=0x7f8c0c1e31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.117+0000 7f8c01ffb640 1 -- 172.21.15.156:0/651392821 >> v1:172.21.15.156:6789/0 conn(0x7f8c0c0b1730 legacy=0x7f8c0c1d7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.117+0000 7f8c01ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:08:46.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.117+0000 7f8c01ffb640 1 monclient: found mon.b 2024-03-25T18:08:46.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.117+0000 7f8c01ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:46.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.117+0000 7f8c01ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:46.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.117+0000 7f8c01ffb640 1 -- 172.21.15.156:0/651392821 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c0c1d8870 con 0x7f8c0c0b4d60 2024-03-25T18:08:46.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.117+0000 7f8c01ffb640 10 monclient: _finish_auth 0 2024-03-25T18:08:46.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.117+0000 7f8c01ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:08:46.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.117+0000 7f8c01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:46.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.117+0000 7f8c13482640 5 monclient: authenticate success, global_id 4216 2024-03-25T18:08:46.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.117+0000 7f8c13482640 10 monclient: _renew_subs 2024-03-25T18:08:46.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.117+0000 7f8c13482640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:46.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.117+0000 7f8c13482640 1 -- 172.21.15.156:0/651392821 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c0c140db0 con 0x7f8c0c0b4d60 2024-03-25T18:08:46.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.117+0000 7f8c13482640 10 client.4216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:46.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.117+0000 7f8c13482640 10 monclient: _renew_subs 2024-03-25T18:08:46.122 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.117+0000 7f8c13482640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:46.122 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.117+0000 7f8c13482640 1 -- 172.21.15.156:0/651392821 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c0c143640 con 0x7f8c0c0b4d60 2024-03-25T18:08:46.122 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.117+0000 7f8c13482640 1 librados: init done 2024-03-25T18:08:46.122 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.119+0000 7f8c13482640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:46.122 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.119+0000 7f8c13482640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:46.122 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.119+0000 7f8c13482640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:46.122 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.119+0000 7f8c13482640 1 -- 172.21.15.156:0/651392821 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c0c0627b0 con 0x7f8c0c0b4d60 2024-03-25T18:08:46.331 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.331+0000 7f8c01ffb640 1 -- 172.21.15.156:0/651392821 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8bfc003610 con 0x7f8c0c0b4d60 2024-03-25T18:08:46.332 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.331+0000 7f8c01ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:46.332 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.331+0000 7f8c01ffb640 1 -- 172.21.15.156:0/651392821 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f8bfc001d80 con 0x7f8c0c0b4d60 2024-03-25T18:08:46.332 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.331+0000 7f8c01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:46.332 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.331+0000 7f8c01ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:46.332 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.331+0000 7f8c01ffb640 10 monclient: dump: 2024-03-25T18:08:46.332 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:08:46.332 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:46.332 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:46.332 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:46.332 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:46.332 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:46.332 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:46.332 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:46.332 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:46.332 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:46.332 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.331+0000 7f8c037fe640 4 set_mon_vals no callback set 2024-03-25T18:08:46.406 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.405+0000 7f8c01ffb640 1 -- 172.21.15.156:0/651392821 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f8bfc003ae0 con 0x7f8c0c0b4d60 2024-03-25T18:08:46.406 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.406+0000 7f8c1121e640 10 monclient: get_auth_request con 0x7f8bf006e270 auth_method 0 2024-03-25T18:08:46.406 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.406+0000 7f8c01ffb640 10 client.4216.objecter ms_handle_connect 0x7f8bf006e270 2024-03-25T18:08:46.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.408+0000 7f8c01ffb640 1 -- 172.21.15.156:0/651392821 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f8bfc0535e0 con 0x7f8c0c0b4d60 2024-03-25T18:08:46.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.408+0000 7f8c01ffb640 10 client.4216.objecter ms_dispatch 0x7f8c0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:46.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.408+0000 7f8c01ffb640 3 client.4216.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:46.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.408+0000 7f8c01ffb640 3 client.4216.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:46.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.408+0000 7f8c01ffb640 20 client.4216.objecter dump_active .. 0 homeless 2024-03-25T18:08:46.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.411+0000 7f8c01ffb640 1 -- 172.21.15.156:0/651392821 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8bfc082e60 con 0x7f8c0c0b4d60 2024-03-25T18:08:46.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.411+0000 7f8c01ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:46.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.411+0000 7f8c01ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:46.624 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.624+0000 7f8c13482640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:46.624 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.624+0000 7f8c13482640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:46.625 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.624+0000 7f8c13482640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:08:46.625 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.624+0000 7f8c13482640 1 -- 172.21.15.156:0/651392821 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c0c137ba0 con 0x7f8c0c0b4d60 2024-03-25T18:08:46.650 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.649+0000 7f8c01ffb640 1 -- 172.21.15.156:0/651392821 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f8bfc04ef20 con 0x7f8c0c0b4d60 2024-03-25T18:08:46.650 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.649+0000 7f8c01ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:46.650 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.649+0000 7f8c01ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:46.650 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:08:46.650 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:46.653 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.652+0000 7f8c13482640 10 librados: watch_flush enter 2024-03-25T18:08:46.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.652+0000 7f8c13482640 10 librados: watch_flush exit 2024-03-25T18:08:46.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.652+0000 7f8c13482640 20 client.4216.objecter shutdown clearing up homeless session... 2024-03-25T18:08:46.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.652+0000 7f8c13482640 10 client.4216.objecter successfully canceled tick 2024-03-25T18:08:46.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.652+0000 7f8c13482640 1 -- 172.21.15.156:0/651392821 >> v1:172.21.15.194:6800/2946427713 conn(0x7f8bf006e270 legacy=0x7f8bf0070730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.652+0000 7f8c13482640 10 monclient: shutdown 2024-03-25T18:08:46.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.652+0000 7f8c13482640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:46.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.652+0000 7f8c13482640 1 -- 172.21.15.156:0/651392821 >> v1:172.21.15.194:6789/0 conn(0x7f8c0c0b4d60 legacy=0x7f8c0c142d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.653+0000 7f8c13482640 1 -- 172.21.15.156:0/651392821 shutdown_connections 2024-03-25T18:08:46.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.653+0000 7f8c13482640 1 -- 172.21.15.156:0/651392821 >> 172.21.15.156:0/651392821 conn(0x7f8c0c0617a0 msgr2=0x7f8c0c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:46.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.653+0000 7f8c13482640 1 -- 172.21.15.156:0/651392821 shutdown_connections 2024-03-25T18:08:46.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.653+0000 7f8c13482640 1 -- 172.21.15.156:0/651392821 wait complete. 2024-03-25T18:08:46.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:46.653+0000 7f8c13482640 1 librados: shutdown 2024-03-25T18:08:46.667 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:47.794 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:26.001 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:48.041 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:26.247 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:48.188 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:26.393 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:48.976 INFO:tasks.ceph.osd.1.smithi156.stderr:==00:00:00:27.181 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:49.178 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:27.377 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:49.303 INFO:tasks.ceph.osd.2.smithi156.stderr:==00:00:00:27.493 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:49.626 INFO:tasks.ceph.osd.3.smithi156.stderr:==00:00:00:27.806 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:49.971 INFO:tasks.ceph.osd.0.smithi156.stderr:==00:00:00:28.162 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:53.474 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:53.563 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.562+0000 7f75323e8640 10 monclient: get_monmap_and_config 2024-03-25T18:08:53.563 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.562+0000 7f75323e8640 10 monclient: build_initial_monmap 2024-03-25T18:08:53.563 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.562+0000 7f75323e8640 10 monclient: monmap: 2024-03-25T18:08:53.563 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:08:53.563 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:53.564 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:08:53.563577+0000 2024-03-25T18:08:53.564 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:08:53.563577+0000 2024-03-25T18:08:53.564 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:53.564 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:53.564 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:53.564 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:08:53.564 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:08:53.564 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:53.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.563+0000 7f75323e8640 1 Processor -- start 2024-03-25T18:08:53.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.563+0000 7f75323e8640 1 -- start start 2024-03-25T18:08:53.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.563+0000 7f75323e8640 10 monclient: init 2024-03-25T18:08:53.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.564+0000 7f75323e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:53.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.564+0000 7f75323e8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:08:53.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.564+0000 7f75323e8640 10 monclient(hunting): picked mon.noname-a con 0x7f752c0b4d60 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:53.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.564+0000 7f75323e8640 10 monclient(hunting): picked mon.noname-c con 0x7f752c0b1730 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:53.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.564+0000 7f75323e8640 10 monclient(hunting): picked mon.noname-b con 0x7f752c14be90 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:53.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.564+0000 7f75323e8640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f752c0af940 con 0x7f752c0b4d60 2024-03-25T18:08:53.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.564+0000 7f75323e8640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f752c0b0a70 con 0x7f752c0b1730 2024-03-25T18:08:53.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.564+0000 7f75323e8640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f752c0b0d50 con 0x7f752c14be90 2024-03-25T18:08:53.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.564+0000 7f75323e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:53.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.564+0000 7f75323e8640 10 monclient(hunting): authenticate will time out at 7014.874953s 2024-03-25T18:08:53.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.564+0000 7f752b7fe640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f752c0b1730 0x7f752c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:57632/0 (socket says 172.21.15.156:57632) 2024-03-25T18:08:53.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.564+0000 7f752b7fe640 1 -- 172.21.15.156:0/1540211243 learned_addr learned my addr 172.21.15.156:0/1540211243 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:08:53.568 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.567+0000 7f752affd640 1 -- 172.21.15.156:0/1540211243 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2027481050 0 0) 0x7f752c0b0a70 con 0x7f752c0b1730 2024-03-25T18:08:53.568 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.567+0000 7f752affd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:53.568 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.567+0000 7f752affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:53.568 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.567+0000 7f752affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:53.568 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.567+0000 7f752affd640 10 monclient(hunting): my global_id is 4219 2024-03-25T18:08:53.568 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.567+0000 7f752affd640 1 -- 172.21.15.156:0/1540211243 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7514003520 con 0x7f752c0b1730 2024-03-25T18:08:53.570 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.568+0000 7f752affd640 1 -- 172.21.15.156:0/1540211243 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1817372913 0 0) 0x7f752c0b0d50 con 0x7f752c14be90 2024-03-25T18:08:53.570 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.568+0000 7f752affd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:53.570 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.568+0000 7f752affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:53.570 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.568+0000 7f752affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:53.570 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.568+0000 7f752affd640 10 monclient(hunting): my global_id is 4229 2024-03-25T18:08:53.570 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.568+0000 7f752affd640 1 -- 172.21.15.156:0/1540211243 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f752c0b0a70 con 0x7f752c14be90 2024-03-25T18:08:53.570 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.569+0000 7f752affd640 1 -- 172.21.15.156:0/1540211243 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 526530435 0 0) 0x7f752c0af940 con 0x7f752c0b4d60 2024-03-25T18:08:53.570 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.569+0000 7f752affd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:53.571 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.569+0000 7f752affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:53.571 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.569+0000 7f752affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:53.571 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.569+0000 7f752affd640 10 monclient(hunting): my global_id is 4221 2024-03-25T18:08:53.571 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.569+0000 7f752affd640 1 -- 172.21.15.156:0/1540211243 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f752c0b0d50 con 0x7f752c0b4d60 2024-03-25T18:08:53.574 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.572+0000 7f752affd640 1 -- 172.21.15.156:0/1540211243 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1566319746 0 0) 0x7f7514003520 con 0x7f752c0b1730 2024-03-25T18:08:53.575 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.572+0000 7f752affd640 1 -- 172.21.15.156:0/1540211243 >> v1:172.21.15.156:6790/0 conn(0x7f752c14be90 legacy=0x7f752c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:53.575 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.572+0000 7f752affd640 1 -- 172.21.15.156:0/1540211243 >> v1:172.21.15.156:6789/0 conn(0x7f752c0b4d60 legacy=0x7f752c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:53.575 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.573+0000 7f752affd640 10 monclient: _finish_hunting 0 2024-03-25T18:08:53.575 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.573+0000 7f752affd640 1 monclient: found mon.noname-c 2024-03-25T18:08:53.575 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.573+0000 7f752affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:53.575 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.573+0000 7f752affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:08:53.575 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.573+0000 7f752affd640 1 -- 172.21.15.156:0/1540211243 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f752c0b1030 con 0x7f752c0b1730 2024-03-25T18:08:53.575 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.573+0000 7f752affd640 10 monclient: _finish_auth 0 2024-03-25T18:08:53.575 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.573+0000 7f752affd640 10 monclient: _check_auth_tickets 2024-03-25T18:08:53.575 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.573+0000 7f752affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:53.575 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.573+0000 7f752affd640 1 -- 172.21.15.156:0/1540211243 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7520001690 con 0x7f752c0b1730 2024-03-25T18:08:53.575 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.573+0000 7f75323e8640 5 monclient: authenticate success, global_id 4219 2024-03-25T18:08:53.575 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.573+0000 7f75323e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:53.575 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.573+0000 7f752affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:53.576 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.573+0000 7f752affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:53.576 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.573+0000 7f752affd640 10 monclient: dump: 2024-03-25T18:08:53.576 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:08:53.576 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:53.576 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:53.576 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:53.576 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:53.576 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:53.576 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:53.576 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:53.576 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:53.576 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:53.576 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.573+0000 7f75323e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:53.578 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.575+0000 7f752affd640 1 -- 172.21.15.156:0/1540211243 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75200018b0 con 0x7f752c0b1730 2024-03-25T18:08:53.578 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.575+0000 7f752affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:53.578 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.575+0000 7f752affd640 1 -- 172.21.15.156:0/1540211243 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7520001c40 con 0x7f752c0b1730 2024-03-25T18:08:53.578 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.575+0000 7f75323e8640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:53.578 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.575+0000 7f75323e8640 4 set_mon_vals no callback set 2024-03-25T18:08:53.578 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.576+0000 7f752affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:53.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.576+0000 7f752affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:53.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.576+0000 7f752affd640 10 monclient: dump: 2024-03-25T18:08:53.579 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:08:53.579 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:53.579 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:53.579 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:53.579 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:53.579 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:53.579 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:53.579 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:53.579 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:53.579 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:53.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.576+0000 7f75323e8640 10 monclient: shutdown 2024-03-25T18:08:53.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.576+0000 7f75323e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:53.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.576+0000 7f75323e8640 1 -- 172.21.15.156:0/1540211243 >> v1:172.21.15.194:6789/0 conn(0x7f752c0b1730 legacy=0x7f752c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:53.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.576+0000 7f75323e8640 1 -- 172.21.15.156:0/1540211243 shutdown_connections 2024-03-25T18:08:53.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.576+0000 7f75323e8640 1 -- 172.21.15.156:0/1540211243 >> 172.21.15.156:0/1540211243 conn(0x7f752c0617a0 msgr2=0x7f752c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:53.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.578+0000 7f75323e8640 1 -- 172.21.15.156:0/1540211243 shutdown_connections 2024-03-25T18:08:53.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.578+0000 7f75323e8640 1 -- 172.21.15.156:0/1540211243 wait complete. 2024-03-25T18:08:53.582 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.578+0000 7f75323e8640 10 monclient: build_initial_monmap 2024-03-25T18:08:53.582 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.578+0000 7f75323e8640 10 monclient: monmap: 2024-03-25T18:08:53.582 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:08:53.582 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:53.582 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:08:53.582 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:08:53.582 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:53.582 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:08:53.582 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:08:53.582 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:08:53.582 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:08:53.582 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:53.582 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.579+0000 7f75323e8640 1 librados: starting msgr at 2024-03-25T18:08:53.582 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.579+0000 7f75323e8640 1 librados: starting objecter 2024-03-25T18:08:53.583 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.579+0000 7f75323e8640 1 Processor -- start 2024-03-25T18:08:53.583 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.579+0000 7f75323e8640 1 -- start start 2024-03-25T18:08:53.583 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.579+0000 7f75323e8640 1 librados: setting wanted keys 2024-03-25T18:08:53.583 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.579+0000 7f75323e8640 1 librados: calling monclient init 2024-03-25T18:08:53.583 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.579+0000 7f75323e8640 10 monclient: init 2024-03-25T18:08:53.583 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.579+0000 7f75323e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:53.583 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.579+0000 7f75323e8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:08:53.583 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.579+0000 7f75323e8640 10 monclient(hunting): picked mon.noname-a con 0x7f752c0b1730 addr v1:172.21.15.156:6789/0 2024-03-25T18:08:53.583 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.579+0000 7f75323e8640 10 monclient(hunting): picked mon.noname-c con 0x7f752c0b4d60 addr v1:172.21.15.156:6790/0 2024-03-25T18:08:53.583 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.580+0000 7f75323e8640 10 monclient(hunting): picked mon.noname-b con 0x7f752c14be90 addr v1:172.21.15.194:6789/0 2024-03-25T18:08:53.583 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.580+0000 7f75323e8640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f752c13a830 con 0x7f752c0b1730 2024-03-25T18:08:53.583 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.580+0000 7f75323e8640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f752c1d7620 con 0x7f752c14be90 2024-03-25T18:08:53.583 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.580+0000 7f75323e8640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f752c1d7900 con 0x7f752c0b4d60 2024-03-25T18:08:53.583 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.580+0000 7f75323e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:53.583 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.580+0000 7f75323e8640 10 monclient(hunting): authenticate will time out at 7014.890917s 2024-03-25T18:08:53.584 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.580+0000 7f7530985640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f752c14be90 0x7f752c1e2ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:57648/0 (socket says 172.21.15.156:57648) 2024-03-25T18:08:53.584 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.580+0000 7f7530985640 1 -- 172.21.15.156:0/4207724483 learned_addr learned my addr 172.21.15.156:0/4207724483 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:08:53.584 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.580+0000 7f7528ff9640 10 client.?.objecter ms_handle_connect 0x7f752c14be90 2024-03-25T18:08:53.584 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.580+0000 7f7528ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:53.584 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.580+0000 7f7528ff9640 10 client.?.objecter ms_handle_connect 0x7f752c0b4d60 2024-03-25T18:08:53.584 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.580+0000 7f7528ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:53.584 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.580+0000 7f7528ff9640 10 client.?.objecter ms_handle_connect 0x7f752c0b1730 2024-03-25T18:08:53.584 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.580+0000 7f7528ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:53.586 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.583+0000 7f7528ff9640 1 -- 172.21.15.156:0/4207724483 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442220370 0 0) 0x7f752c1d7900 con 0x7f752c0b4d60 2024-03-25T18:08:53.586 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.583+0000 7f7528ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:53.586 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.583+0000 7f7528ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:53.586 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.583+0000 7f7528ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:53.586 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.583+0000 7f7528ff9640 10 monclient(hunting): my global_id is 4232 2024-03-25T18:08:53.586 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.583+0000 7f7528ff9640 1 -- 172.21.15.156:0/4207724483 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7518004eb0 con 0x7f752c0b4d60 2024-03-25T18:08:53.586 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.583+0000 7f7528ff9640 1 -- 172.21.15.156:0/4207724483 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854971426 0 0) 0x7f752c1d7620 con 0x7f752c14be90 2024-03-25T18:08:53.586 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.583+0000 7f7528ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:53.586 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.583+0000 7f7528ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:53.587 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.583+0000 7f7528ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:53.587 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.583+0000 7f7528ff9640 10 monclient(hunting): my global_id is 4222 2024-03-25T18:08:53.587 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.583+0000 7f7528ff9640 1 -- 172.21.15.156:0/4207724483 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f752c1d7900 con 0x7f752c14be90 2024-03-25T18:08:53.587 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.583+0000 7f7528ff9640 1 -- 172.21.15.156:0/4207724483 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1412695202 0 0) 0x7f752c13a830 con 0x7f752c0b1730 2024-03-25T18:08:53.587 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.583+0000 7f7528ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:53.587 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.583+0000 7f7528ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:53.587 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.583+0000 7f7528ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:53.587 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.583+0000 7f7528ff9640 10 monclient(hunting): my global_id is 4224 2024-03-25T18:08:53.587 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.583+0000 7f7528ff9640 1 -- 172.21.15.156:0/4207724483 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f752c1d7620 con 0x7f752c0b1730 2024-03-25T18:08:53.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.587+0000 7f7528ff9640 1 -- 172.21.15.156:0/4207724483 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1972536480 0 0) 0x7f7518004eb0 con 0x7f752c0b4d60 2024-03-25T18:08:53.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.588+0000 7f7528ff9640 1 -- 172.21.15.156:0/4207724483 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f752c13a830 con 0x7f752c0b4d60 2024-03-25T18:08:53.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.588+0000 7f7528ff9640 1 -- 172.21.15.156:0/4207724483 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7520001630 con 0x7f752c0b4d60 2024-03-25T18:08:53.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.588+0000 7f7528ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:53.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.588+0000 7f7528ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:53.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.588+0000 7f7528ff9640 10 monclient(hunting): dump: 2024-03-25T18:08:53.589 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:08:53.589 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:53.589 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:53.589 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:53.589 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:53.590 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:53.590 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:53.590 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:53.590 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:53.590 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:53.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.588+0000 7f7528ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:53.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.590+0000 7f7528ff9640 1 -- 172.21.15.156:0/4207724483 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4272891345 0 0) 0x7f752c1d7900 con 0x7f752c14be90 2024-03-25T18:08:53.591 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.590+0000 7f7528ff9640 1 -- 172.21.15.156:0/4207724483 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7518004eb0 con 0x7f752c14be90 2024-03-25T18:08:53.591 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.590+0000 7f7528ff9640 1 -- 172.21.15.156:0/4207724483 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7524001c10 con 0x7f752c14be90 2024-03-25T18:08:53.591 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.590+0000 7f7528ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:53.591 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.590+0000 7f7528ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:53.591 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.590+0000 7f7528ff9640 10 monclient(hunting): dump: 2024-03-25T18:08:53.591 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:08:53.591 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:53.591 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:53.591 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:53.591 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:53.591 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:53.591 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:53.591 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:53.592 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:53.592 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:53.592 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.591+0000 7f7528ff9640 1 -- 172.21.15.156:0/4207724483 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2070248601 0 0) 0x7f752c1d7620 con 0x7f752c0b1730 2024-03-25T18:08:53.592 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.591+0000 7f7528ff9640 1 -- 172.21.15.156:0/4207724483 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f752c1d7900 con 0x7f752c0b1730 2024-03-25T18:08:53.592 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.591+0000 7f7528ff9640 1 -- 172.21.15.156:0/4207724483 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f751c001830 con 0x7f752c0b1730 2024-03-25T18:08:53.592 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.591+0000 7f7528ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:53.592 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.591+0000 7f7528ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:53.593 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.591+0000 7f7528ff9640 10 monclient(hunting): dump: 2024-03-25T18:08:53.593 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:08:53.593 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:53.593 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:53.593 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:53.593 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:53.593 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:53.593 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:53.593 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:53.593 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:53.593 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:53.594 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.592+0000 7f7528ff9640 1 -- 172.21.15.156:0/4207724483 <== mon.2 v1:172.21.15.156:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3961622968 0 0) 0x7f752c13a830 con 0x7f752c0b4d60 2024-03-25T18:08:53.594 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.592+0000 7f7528ff9640 1 -- 172.21.15.156:0/4207724483 >> v1:172.21.15.194:6789/0 conn(0x7f752c14be90 legacy=0x7f752c1e2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:53.594 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.592+0000 7f7528ff9640 1 -- 172.21.15.156:0/4207724483 >> v1:172.21.15.156:6789/0 conn(0x7f752c0b1730 legacy=0x7f752c1d7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:53.594 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.592+0000 7f7528ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:08:53.594 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.592+0000 7f7528ff9640 1 monclient: found mon.c 2024-03-25T18:08:53.594 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.592+0000 7f7528ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:53.594 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.592+0000 7f7528ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:08:53.594 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.592+0000 7f7528ff9640 1 -- 172.21.15.156:0/4207724483 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f752c142820 con 0x7f752c0b4d60 2024-03-25T18:08:53.594 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.592+0000 7f7528ff9640 10 monclient: _finish_auth 0 2024-03-25T18:08:53.594 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.592+0000 7f7528ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:08:53.595 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.592+0000 7f7528ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:53.595 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.592+0000 7f75323e8640 5 monclient: authenticate success, global_id 4232 2024-03-25T18:08:53.595 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.592+0000 7f75323e8640 10 monclient: _renew_subs 2024-03-25T18:08:53.595 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.592+0000 7f75323e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:08:53.595 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.592+0000 7f75323e8640 1 -- 172.21.15.156:0/4207724483 --> v1:172.21.15.156:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f752c13bcf0 con 0x7f752c0b4d60 2024-03-25T18:08:53.595 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.593+0000 7f75323e8640 10 client.4232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:53.595 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.593+0000 7f75323e8640 10 monclient: _renew_subs 2024-03-25T18:08:53.595 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.593+0000 7f75323e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:08:53.595 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.593+0000 7f75323e8640 1 -- 172.21.15.156:0/4207724483 --> v1:172.21.15.156:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f752c143280 con 0x7f752c0b4d60 2024-03-25T18:08:53.595 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.593+0000 7f75323e8640 1 librados: init done 2024-03-25T18:08:53.596 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.594+0000 7f75323e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:53.596 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.594+0000 7f75323e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:53.596 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.594+0000 7f75323e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:08:53.596 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.594+0000 7f75323e8640 1 -- 172.21.15.156:0/4207724483 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f752c1d7a90 con 0x7f752c0b4d60 2024-03-25T18:08:53.902 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.901+0000 7f7528ff9640 1 -- 172.21.15.156:0/4207724483 <== mon.2 v1:172.21.15.156:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7520004a90 con 0x7f752c0b4d60 2024-03-25T18:08:53.903 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.902+0000 7f7528ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:53.903 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.902+0000 7f7528ff9640 1 -- 172.21.15.156:0/4207724483 <== mon.2 v1:172.21.15.156:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7520004da0 con 0x7f752c0b4d60 2024-03-25T18:08:53.903 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.902+0000 7f7528ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:53.903 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.902+0000 7f7528ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:53.903 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.902+0000 7f7528ff9640 10 monclient: dump: 2024-03-25T18:08:53.903 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:08:53.903 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:08:53.903 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:53.903 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:08:53.903 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:53.903 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:08:53.904 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:08:53.904 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:08:53.904 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:08:53.904 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:08:53.904 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.902+0000 7f752a7fc640 4 set_mon_vals no callback set 2024-03-25T18:08:53.904 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.903+0000 7f7528ff9640 1 -- 172.21.15.156:0/4207724483 <== mon.2 v1:172.21.15.156:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f752001ad70 con 0x7f752c0b4d60 2024-03-25T18:08:53.904 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.903+0000 7f752bfff640 10 monclient: get_auth_request con 0x7f751806e0c0 auth_method 0 2024-03-25T18:08:53.905 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.904+0000 7f7528ff9640 10 client.4232.objecter ms_handle_connect 0x7f751806e0c0 2024-03-25T18:08:53.911 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.910+0000 7f7528ff9640 1 -- 172.21.15.156:0/4207724483 <== mon.2 v1:172.21.15.156:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f75200548e0 con 0x7f752c0b4d60 2024-03-25T18:08:53.911 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.910+0000 7f7528ff9640 10 client.4232.objecter ms_dispatch 0x7f752c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:53.911 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.910+0000 7f7528ff9640 3 client.4232.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:53.911 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.910+0000 7f7528ff9640 3 client.4232.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:53.911 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.910+0000 7f7528ff9640 20 client.4232.objecter dump_active .. 0 homeless 2024-03-25T18:08:53.911 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.910+0000 7f7528ff9640 1 -- 172.21.15.156:0/4207724483 <== mon.2 v1:172.21.15.156:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f75200503f0 con 0x7f752c0b4d60 2024-03-25T18:08:53.911 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.910+0000 7f7528ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:53.911 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:53.910+0000 7f7528ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:54.091 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:54.091+0000 7f75323e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:54.092 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:54.091+0000 7f75323e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:54.092 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:54.091+0000 7f75323e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:08:54.092 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:54.091+0000 7f75323e8640 1 -- 172.21.15.156:0/4207724483 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f752c0627b0 con 0x7f752c0b4d60 2024-03-25T18:08:54.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:54.152+0000 7f7528ff9640 1 -- 172.21.15.156:0/4207724483 <== mon.2 v1:172.21.15.156:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f7520052320 con 0x7f752c0b4d60 2024-03-25T18:08:54.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:54.152+0000 7f7528ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:54.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:54.152+0000 7f7528ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:54.153 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:08:54.153 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:54.156 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:54.154+0000 7f75323e8640 10 librados: watch_flush enter 2024-03-25T18:08:54.156 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:54.154+0000 7f75323e8640 10 librados: watch_flush exit 2024-03-25T18:08:54.156 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:54.154+0000 7f75323e8640 20 client.4232.objecter shutdown clearing up homeless session... 2024-03-25T18:08:54.156 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:54.154+0000 7f75323e8640 10 client.4232.objecter successfully canceled tick 2024-03-25T18:08:54.156 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:54.155+0000 7f75323e8640 1 -- 172.21.15.156:0/4207724483 >> v1:172.21.15.194:6800/2946427713 conn(0x7f751806e0c0 legacy=0x7f7518070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:54.156 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:54.155+0000 7f75323e8640 10 monclient: shutdown 2024-03-25T18:08:54.157 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:54.155+0000 7f75323e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:54.157 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:54.155+0000 7f75323e8640 1 -- 172.21.15.156:0/4207724483 >> v1:172.21.15.156:6790/0 conn(0x7f752c0b4d60 legacy=0x7f752c13b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:54.157 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:54.155+0000 7f75323e8640 1 -- 172.21.15.156:0/4207724483 shutdown_connections 2024-03-25T18:08:54.157 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:54.155+0000 7f75323e8640 1 -- 172.21.15.156:0/4207724483 >> 172.21.15.156:0/4207724483 conn(0x7f752c0617a0 msgr2=0x7f752c0b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:54.157 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:54.155+0000 7f75323e8640 1 -- 172.21.15.156:0/4207724483 shutdown_connections 2024-03-25T18:08:54.157 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:54.155+0000 7f75323e8640 1 -- 172.21.15.156:0/4207724483 wait complete. 2024-03-25T18:08:54.157 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:08:54.155+0000 7f75323e8640 1 librados: shutdown 2024-03-25T18:08:54.170 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:56.328 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-03-25T18:08:56.324+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T18:08:56.370 INFO:tasks.ceph.osd.4.smithi194.stderr:2024-03-25T18:08:56.312+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T18:08:56.706 INFO:tasks.ceph.osd.6.smithi194.stderr:2024-03-25T18:08:56.703+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T18:08:57.356 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-03-25T18:08:57.347+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T18:08:58.133 INFO:tasks.ceph.osd.1.smithi156.stderr:2024-03-25T18:08:58.129+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T18:08:58.587 INFO:tasks.ceph.osd.2.smithi156.stderr:2024-03-25T18:08:58.582+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T18:08:59.028 INFO:tasks.ceph.osd.3.smithi156.stderr:2024-03-25T18:08:59.025+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T18:08:59.354 INFO:tasks.ceph.osd.0.smithi156.stderr:2024-03-25T18:08:59.350+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T18:09:00.976 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:01.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.092+0000 7fbc77d7b640 10 monclient: get_monmap_and_config 2024-03-25T18:09:01.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.092+0000 7fbc77d7b640 10 monclient: build_initial_monmap 2024-03-25T18:09:01.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.092+0000 7fbc77d7b640 10 monclient: monmap: 2024-03-25T18:09:01.093 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:09:01.093 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:01.093 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:09:01.093423+0000 2024-03-25T18:09:01.093 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:09:01.093423+0000 2024-03-25T18:09:01.093 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:01.093 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:01.093 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:09:01.094 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:09:01.094 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:09:01.094 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:01.095 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.093+0000 7fbc77d7b640 1 Processor -- start 2024-03-25T18:09:01.095 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.093+0000 7fbc77d7b640 1 -- start start 2024-03-25T18:09:01.095 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.093+0000 7fbc77d7b640 10 monclient: init 2024-03-25T18:09:01.095 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.094+0000 7fbc77d7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:01.095 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.094+0000 7fbc77d7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:09:01.095 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.094+0000 7fbc77d7b640 10 monclient(hunting): picked mon.noname-a con 0x7fbc700a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:09:01.095 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.094+0000 7fbc77d7b640 10 monclient(hunting): picked mon.noname-b con 0x7fbc7013f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:09:01.095 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.094+0000 7fbc77d7b640 10 monclient(hunting): picked mon.noname-c con 0x7fbc701430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:09:01.095 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.094+0000 7fbc77d7b640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc70142520 con 0x7fbc700a98f0 2024-03-25T18:09:01.095 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.094+0000 7fbc77d7b640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc700b28a0 con 0x7fbc701430c0 2024-03-25T18:09:01.096 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.094+0000 7fbc77d7b640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc700b2a30 con 0x7fbc7013f710 2024-03-25T18:09:01.096 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.094+0000 7fbc77d7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:01.096 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.094+0000 7fbc77d7b640 10 monclient(hunting): authenticate will time out at 7022.405028s 2024-03-25T18:09:01.104 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.103+0000 7fbc75b17640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fbc700a98f0 0x7fbc700a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:56582/0 (socket says 172.21.15.156:56582) 2024-03-25T18:09:01.104 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.103+0000 7fbc75b17640 1 -- 172.21.15.156:0/3078588727 learned_addr learned my addr 172.21.15.156:0/3078588727 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:09:01.125 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.124+0000 7fbc74b15640 1 -- 172.21.15.156:0/3078588727 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2602301226 0 0) 0x7fbc700b28a0 con 0x7fbc701430c0 2024-03-25T18:09:01.125 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.124+0000 7fbc74b15640 10 monclient(hunting): _negotiate 2024-03-25T18:09:01.125 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.124+0000 7fbc74b15640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:01.125 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.124+0000 7fbc74b15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:01.125 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.124+0000 7fbc74b15640 10 monclient(hunting): my global_id is 4249 2024-03-25T18:09:01.125 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.124+0000 7fbc74b15640 1 -- 172.21.15.156:0/3078588727 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc5c003520 con 0x7fbc701430c0 2024-03-25T18:09:01.136 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.134+0000 7fbc74b15640 1 -- 172.21.15.156:0/3078588727 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2748127321 0 0) 0x7fbc700b2a30 con 0x7fbc7013f710 2024-03-25T18:09:01.136 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.135+0000 7fbc74b15640 10 monclient(hunting): _negotiate 2024-03-25T18:09:01.136 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.135+0000 7fbc74b15640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:01.136 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.135+0000 7fbc74b15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:01.136 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.135+0000 7fbc74b15640 10 monclient(hunting): my global_id is 4259 2024-03-25T18:09:01.136 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.135+0000 7fbc74b15640 1 -- 172.21.15.156:0/3078588727 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc700b28a0 con 0x7fbc7013f710 2024-03-25T18:09:01.136 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.135+0000 7fbc74b15640 1 -- 172.21.15.156:0/3078588727 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 115449029 0 0) 0x7fbc5c003520 con 0x7fbc701430c0 2024-03-25T18:09:01.136 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.135+0000 7fbc74b15640 1 -- 172.21.15.156:0/3078588727 >> v1:172.21.15.156:6790/0 conn(0x7fbc7013f710 legacy=0x7fbc701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:01.136 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.135+0000 7fbc74b15640 1 -- 172.21.15.156:0/3078588727 >> v1:172.21.15.156:6789/0 conn(0x7fbc700a98f0 legacy=0x7fbc700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:01.136 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.135+0000 7fbc74b15640 10 monclient: _finish_hunting 0 2024-03-25T18:09:01.136 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.135+0000 7fbc74b15640 1 monclient: found mon.noname-c 2024-03-25T18:09:01.136 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.135+0000 7fbc74b15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:01.137 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.135+0000 7fbc74b15640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:09:01.137 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.135+0000 7fbc74b15640 1 -- 172.21.15.156:0/3078588727 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc700b44c0 con 0x7fbc701430c0 2024-03-25T18:09:01.137 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.135+0000 7fbc74b15640 10 monclient: _finish_auth 0 2024-03-25T18:09:01.137 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.135+0000 7fbc74b15640 10 monclient: _check_auth_tickets 2024-03-25T18:09:01.137 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.135+0000 7fbc74b15640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:01.137 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.135+0000 7fbc77d7b640 5 monclient: authenticate success, global_id 4249 2024-03-25T18:09:01.137 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.135+0000 7fbc77d7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:01.148 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.148+0000 7fbc74b15640 1 -- 172.21.15.156:0/3078588727 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fbc680019d0 con 0x7fbc701430c0 2024-03-25T18:09:01.148 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.148+0000 7fbc74b15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:01.148 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.148+0000 7fbc74b15640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:01.148 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.148+0000 7fbc74b15640 10 monclient: dump: 2024-03-25T18:09:01.149 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:01.149 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:01.149 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:01.149 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:01.149 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:01.149 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:01.149 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:01.149 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:01.149 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:01.149 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:01.149 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.148+0000 7fbc74b15640 1 -- 172.21.15.156:0/3078588727 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc68001c20 con 0x7fbc701430c0 2024-03-25T18:09:01.149 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.148+0000 7fbc77d7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:01.149 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.148+0000 7fbc74b15640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:01.149 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.148+0000 7fbc77d7b640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:01.149 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.148+0000 7fbc74b15640 1 -- 172.21.15.156:0/3078588727 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fbc68001fb0 con 0x7fbc701430c0 2024-03-25T18:09:01.150 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.148+0000 7fbc77d7b640 4 set_mon_vals no callback set 2024-03-25T18:09:01.150 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.148+0000 7fbc74b15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:01.150 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.148+0000 7fbc74b15640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:01.150 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.148+0000 7fbc74b15640 10 monclient: dump: 2024-03-25T18:09:01.150 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:01.150 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:01.150 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:01.150 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:01.150 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:01.150 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:01.150 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:01.150 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:01.150 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:01.150 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:01.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.148+0000 7fbc77d7b640 10 monclient: shutdown 2024-03-25T18:09:01.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.148+0000 7fbc77d7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:01.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.148+0000 7fbc77d7b640 1 -- 172.21.15.156:0/3078588727 >> v1:172.21.15.194:6789/0 conn(0x7fbc701430c0 legacy=0x7fbc7014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:01.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.149+0000 7fbc77d7b640 1 -- 172.21.15.156:0/3078588727 shutdown_connections 2024-03-25T18:09:01.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.149+0000 7fbc77d7b640 1 -- 172.21.15.156:0/3078588727 >> 172.21.15.156:0/3078588727 conn(0x7fbc700b7ea0 msgr2=0x7fbc700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:01.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.149+0000 7fbc77d7b640 1 -- 172.21.15.156:0/3078588727 shutdown_connections 2024-03-25T18:09:01.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.149+0000 7fbc77d7b640 1 -- 172.21.15.156:0/3078588727 wait complete. 2024-03-25T18:09:01.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.150+0000 7fbc77d7b640 10 monclient: build_initial_monmap 2024-03-25T18:09:01.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.150+0000 7fbc77d7b640 10 monclient: monmap: 2024-03-25T18:09:01.152 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:09:01.153 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:01.153 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:09:01.153 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:09:01.153 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:01.153 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:09:01.153 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:09:01.153 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:09:01.153 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:09:01.153 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:01.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.150+0000 7fbc77d7b640 1 librados: starting msgr at 2024-03-25T18:09:01.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.150+0000 7fbc77d7b640 1 librados: starting objecter 2024-03-25T18:09:01.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.150+0000 7fbc77d7b640 1 Processor -- start 2024-03-25T18:09:01.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.150+0000 7fbc77d7b640 1 -- start start 2024-03-25T18:09:01.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.150+0000 7fbc77d7b640 1 librados: setting wanted keys 2024-03-25T18:09:01.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.150+0000 7fbc77d7b640 1 librados: calling monclient init 2024-03-25T18:09:01.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.150+0000 7fbc77d7b640 10 monclient: init 2024-03-25T18:09:01.154 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.151+0000 7fbc77d7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:01.154 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.151+0000 7fbc77d7b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:09:01.154 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.151+0000 7fbc77d7b640 10 monclient(hunting): picked mon.noname-c con 0x7fbc700a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:09:01.154 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.151+0000 7fbc77d7b640 10 monclient(hunting): picked mon.noname-b con 0x7fbc7013f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:09:01.154 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.151+0000 7fbc77d7b640 10 monclient(hunting): picked mon.noname-a con 0x7fbc701430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:09:01.154 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.151+0000 7fbc77d7b640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc7014c610 con 0x7fbc701430c0 2024-03-25T18:09:01.154 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.151+0000 7fbc77d7b640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc701319b0 con 0x7fbc7013f710 2024-03-25T18:09:01.154 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.151+0000 7fbc77d7b640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc7014dce0 con 0x7fbc700a98f0 2024-03-25T18:09:01.154 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.151+0000 7fbc77d7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:01.154 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.151+0000 7fbc77d7b640 10 monclient(hunting): authenticate will time out at 7022.462116s 2024-03-25T18:09:01.161 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.160+0000 7fbc75b17640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7fbc700a98f0 0x7fbc7014bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:39712/0 (socket says 172.21.15.156:39712) 2024-03-25T18:09:01.161 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.161+0000 7fbc75b17640 1 -- 172.21.15.156:0/1511863331 learned_addr learned my addr 172.21.15.156:0/1511863331 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:09:01.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.162+0000 7fbc667fc640 10 client.?.objecter ms_handle_connect 0x7fbc700a98f0 2024-03-25T18:09:01.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.162+0000 7fbc667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:01.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.168+0000 7fbc667fc640 10 client.?.objecter ms_handle_connect 0x7fbc701430c0 2024-03-25T18:09:01.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.168+0000 7fbc667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:01.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.172+0000 7fbc667fc640 10 client.?.objecter ms_handle_connect 0x7fbc7013f710 2024-03-25T18:09:01.173 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.172+0000 7fbc667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:01.183 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.182+0000 7fbc667fc640 1 -- 172.21.15.156:0/1511863331 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1806305550 0 0) 0x7fbc701319b0 con 0x7fbc7013f710 2024-03-25T18:09:01.183 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.182+0000 7fbc667fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:01.183 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.182+0000 7fbc667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:01.183 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.182+0000 7fbc667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:01.183 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.182+0000 7fbc667fc640 10 monclient(hunting): my global_id is 4252 2024-03-25T18:09:01.183 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.182+0000 7fbc667fc640 1 -- 172.21.15.156:0/1511863331 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc54004eb0 con 0x7fbc7013f710 2024-03-25T18:09:01.188 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.187+0000 7fbc667fc640 1 -- 172.21.15.156:0/1511863331 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168596777 0 0) 0x7fbc7014dce0 con 0x7fbc700a98f0 2024-03-25T18:09:01.188 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.187+0000 7fbc667fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:01.188 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.187+0000 7fbc667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:01.188 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.187+0000 7fbc667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:01.188 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.187+0000 7fbc667fc640 10 monclient(hunting): my global_id is 4262 2024-03-25T18:09:01.188 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.187+0000 7fbc667fc640 1 -- 172.21.15.156:0/1511863331 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc701319b0 con 0x7fbc700a98f0 2024-03-25T18:09:01.193 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.192+0000 7fbc667fc640 1 -- 172.21.15.156:0/1511863331 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807266851 0 0) 0x7fbc7014c610 con 0x7fbc701430c0 2024-03-25T18:09:01.193 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.192+0000 7fbc667fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:01.193 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.192+0000 7fbc667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:01.193 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.192+0000 7fbc667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:01.193 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.192+0000 7fbc667fc640 10 monclient(hunting): my global_id is 4254 2024-03-25T18:09:01.193 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.192+0000 7fbc667fc640 1 -- 172.21.15.156:0/1511863331 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc7014dce0 con 0x7fbc701430c0 2024-03-25T18:09:01.196 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.194+0000 7fbc667fc640 1 -- 172.21.15.156:0/1511863331 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2681016245 0 0) 0x7fbc54004eb0 con 0x7fbc7013f710 2024-03-25T18:09:01.196 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.195+0000 7fbc667fc640 1 -- 172.21.15.156:0/1511863331 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc7014c610 con 0x7fbc7013f710 2024-03-25T18:09:01.196 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.195+0000 7fbc667fc640 1 -- 172.21.15.156:0/1511863331 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fbc60001930 con 0x7fbc7013f710 2024-03-25T18:09:01.196 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.195+0000 7fbc667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:01.196 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.195+0000 7fbc667fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:01.196 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.195+0000 7fbc667fc640 10 monclient(hunting): dump: 2024-03-25T18:09:01.196 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:01.196 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:01.196 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:01.196 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:01.196 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:01.196 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:01.196 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:01.196 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:01.196 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:01.196 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:01.197 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.195+0000 7fbc667fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:01.216 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.215+0000 7fbc667fc640 1 -- 172.21.15.156:0/1511863331 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 759061513 0 0) 0x7fbc7014c610 con 0x7fbc7013f710 2024-03-25T18:09:01.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.215+0000 7fbc667fc640 1 -- 172.21.15.156:0/1511863331 >> v1:172.21.15.156:6790/0 conn(0x7fbc700a98f0 legacy=0x7fbc7014bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:01.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.215+0000 7fbc667fc640 1 -- 172.21.15.156:0/1511863331 >> v1:172.21.15.156:6789/0 conn(0x7fbc701430c0 legacy=0x7fbc701312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:01.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.215+0000 7fbc667fc640 10 monclient: _finish_hunting 0 2024-03-25T18:09:01.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.215+0000 7fbc667fc640 1 monclient: found mon.b 2024-03-25T18:09:01.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.215+0000 7fbc667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:01.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.215+0000 7fbc667fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:09:01.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.215+0000 7fbc667fc640 1 -- 172.21.15.156:0/1511863331 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc7014dfc0 con 0x7fbc7013f710 2024-03-25T18:09:01.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.215+0000 7fbc667fc640 10 monclient: _finish_auth 0 2024-03-25T18:09:01.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.215+0000 7fbc667fc640 10 monclient: _check_auth_tickets 2024-03-25T18:09:01.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.215+0000 7fbc667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:01.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.215+0000 7fbc77d7b640 5 monclient: authenticate success, global_id 4252 2024-03-25T18:09:01.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.215+0000 7fbc77d7b640 10 monclient: _renew_subs 2024-03-25T18:09:01.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.215+0000 7fbc77d7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:09:01.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.215+0000 7fbc77d7b640 1 -- 172.21.15.156:0/1511863331 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc700b7900 con 0x7fbc7013f710 2024-03-25T18:09:01.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.216+0000 7fbc77d7b640 10 client.4252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:01.218 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.216+0000 7fbc77d7b640 10 monclient: _renew_subs 2024-03-25T18:09:01.218 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.216+0000 7fbc77d7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:09:01.218 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.216+0000 7fbc77d7b640 1 -- 172.21.15.156:0/1511863331 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc7014e3e0 con 0x7fbc7013f710 2024-03-25T18:09:01.218 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.216+0000 7fbc77d7b640 1 librados: init done 2024-03-25T18:09:01.218 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.216+0000 7fbc77d7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:01.218 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.216+0000 7fbc77d7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:01.219 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.216+0000 7fbc77d7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:09:01.219 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.216+0000 7fbc77d7b640 1 -- 172.21.15.156:0/1511863331 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc7014e6e0 con 0x7fbc7013f710 2024-03-25T18:09:01.511 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.510+0000 7fbc667fc640 1 -- 172.21.15.156:0/1511863331 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc60001a90 con 0x7fbc7013f710 2024-03-25T18:09:01.511 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.510+0000 7fbc667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:01.511 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.510+0000 7fbc667fc640 1 -- 172.21.15.156:0/1511863331 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fbc600038c0 con 0x7fbc7013f710 2024-03-25T18:09:01.511 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.510+0000 7fbc667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:01.511 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.510+0000 7fbc667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:01.511 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.510+0000 7fbc667fc640 10 monclient: dump: 2024-03-25T18:09:01.511 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:01.511 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:01.512 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:01.512 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:01.512 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:01.512 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:01.512 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:01.512 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:01.512 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:01.512 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:01.512 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.510+0000 7fbc67fff640 4 set_mon_vals no callback set 2024-03-25T18:09:01.512 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.511+0000 7fbc667fc640 1 -- 172.21.15.156:0/1511863331 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7fbc60003ae0 con 0x7fbc7013f710 2024-03-25T18:09:01.513 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.511+0000 7fbc75b17640 10 monclient: get_auth_request con 0x7fbc5406dda0 auth_method 0 2024-03-25T18:09:01.513 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.512+0000 7fbc667fc640 10 client.4252.objecter ms_handle_connect 0x7fbc5406dda0 2024-03-25T18:09:01.516 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.515+0000 7fbc667fc640 1 -- 172.21.15.156:0/1511863331 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7fbc60053660 con 0x7fbc7013f710 2024-03-25T18:09:01.516 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.515+0000 7fbc667fc640 10 client.4252.objecter ms_dispatch 0x7fbc70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:01.516 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.515+0000 7fbc667fc640 3 client.4252.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:01.516 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.515+0000 7fbc667fc640 3 client.4252.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:01.516 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.515+0000 7fbc667fc640 20 client.4252.objecter dump_active .. 0 homeless 2024-03-25T18:09:01.516 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.515+0000 7fbc667fc640 1 -- 172.21.15.156:0/1511863331 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbc6004efa0 con 0x7fbc7013f710 2024-03-25T18:09:01.516 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.515+0000 7fbc667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:01.516 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.515+0000 7fbc667fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:01.790 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.789+0000 7fbc77d7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:01.790 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.789+0000 7fbc77d7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:01.790 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.789+0000 7fbc77d7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:09:01.790 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.789+0000 7fbc77d7b640 1 -- 172.21.15.156:0/1511863331 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc700627b0 con 0x7fbc7013f710 2024-03-25T18:09:01.833 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.832+0000 7fbc667fc640 1 -- 172.21.15.156:0/1511863331 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7fbc600510a0 con 0x7fbc7013f710 2024-03-25T18:09:01.833 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.832+0000 7fbc667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:01.833 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.832+0000 7fbc667fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:01.834 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:09:01.834 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:01.835 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.834+0000 7fbc77d7b640 10 librados: watch_flush enter 2024-03-25T18:09:01.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.834+0000 7fbc77d7b640 10 librados: watch_flush exit 2024-03-25T18:09:01.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.834+0000 7fbc77d7b640 20 client.4252.objecter shutdown clearing up homeless session... 2024-03-25T18:09:01.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.834+0000 7fbc77d7b640 10 client.4252.objecter successfully canceled tick 2024-03-25T18:09:01.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.834+0000 7fbc77d7b640 1 -- 172.21.15.156:0/1511863331 >> v1:172.21.15.194:6800/2946427713 conn(0x7fbc5406dda0 legacy=0x7fbc54070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:01.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.834+0000 7fbc77d7b640 10 monclient: shutdown 2024-03-25T18:09:01.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.834+0000 7fbc77d7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:01.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.834+0000 7fbc77d7b640 1 -- 172.21.15.156:0/1511863331 >> v1:172.21.15.194:6789/0 conn(0x7fbc7013f710 legacy=0x7fbc70130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:01.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.835+0000 7fbc77d7b640 1 -- 172.21.15.156:0/1511863331 shutdown_connections 2024-03-25T18:09:01.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.835+0000 7fbc77d7b640 1 -- 172.21.15.156:0/1511863331 >> 172.21.15.156:0/1511863331 conn(0x7fbc700b7ea0 msgr2=0x7fbc70074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:01.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.835+0000 7fbc77d7b640 1 -- 172.21.15.156:0/1511863331 shutdown_connections 2024-03-25T18:09:01.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.835+0000 7fbc77d7b640 1 -- 172.21.15.156:0/1511863331 wait complete. 2024-03-25T18:09:01.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:01.835+0000 7fbc77d7b640 1 librados: shutdown 2024-03-25T18:09:01.848 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:08.654 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:08.771 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.770+0000 7f827d94d640 10 monclient: get_monmap_and_config 2024-03-25T18:09:08.771 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.770+0000 7f827d94d640 10 monclient: build_initial_monmap 2024-03-25T18:09:08.771 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.770+0000 7f827d94d640 10 monclient: monmap: 2024-03-25T18:09:08.771 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:09:08.772 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:08.772 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:09:08.771631+0000 2024-03-25T18:09:08.772 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:09:08.771631+0000 2024-03-25T18:09:08.772 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:08.772 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:08.772 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:09:08.772 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:09:08.772 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:09:08.772 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:08.772 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.771+0000 7f827d94d640 1 Processor -- start 2024-03-25T18:09:08.773 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.771+0000 7f827d94d640 1 -- start start 2024-03-25T18:09:08.773 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.771+0000 7f827d94d640 10 monclient: init 2024-03-25T18:09:08.773 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.772+0000 7f827d94d640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:08.773 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.772+0000 7f827d94d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:09:08.773 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.772+0000 7f827d94d640 10 monclient(hunting): picked mon.noname-a con 0x7f82780b4d60 addr v1:172.21.15.156:6789/0 2024-03-25T18:09:08.774 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.772+0000 7f827d94d640 10 monclient(hunting): picked mon.noname-b con 0x7f82780b1730 addr v1:172.21.15.156:6790/0 2024-03-25T18:09:08.774 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.772+0000 7f827d94d640 10 monclient(hunting): picked mon.noname-c con 0x7f827814be90 addr v1:172.21.15.194:6789/0 2024-03-25T18:09:08.774 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.772+0000 7f827d94d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82780af940 con 0x7f82780b4d60 2024-03-25T18:09:08.774 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.772+0000 7f827d94d640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82780b0a70 con 0x7f827814be90 2024-03-25T18:09:08.774 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.772+0000 7f827d94d640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82780b0d50 con 0x7f82780b1730 2024-03-25T18:09:08.774 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.772+0000 7f827d94d640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:08.774 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.772+0000 7f827d94d640 10 monclient(hunting): authenticate will time out at 7030.082929s 2024-03-25T18:09:08.779 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.779+0000 7f82777fe640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f827814be90 0x7f827814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:47722/0 (socket says 172.21.15.156:47722) 2024-03-25T18:09:08.779 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.779+0000 7f82777fe640 1 -- 172.21.15.156:0/4038193556 learned_addr learned my addr 172.21.15.156:0/4038193556 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:09:08.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.786+0000 7f8275ffb640 1 -- 172.21.15.156:0/4038193556 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458245561 0 0) 0x7f82780b0a70 con 0x7f827814be90 2024-03-25T18:09:08.787 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.786+0000 7f8275ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:09:08.787 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.786+0000 7f8275ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:08.787 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.786+0000 7f8275ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:08.787 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.786+0000 7f8275ffb640 10 monclient(hunting): my global_id is 4255 2024-03-25T18:09:08.787 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.786+0000 7f8275ffb640 1 -- 172.21.15.156:0/4038193556 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8260003520 con 0x7f827814be90 2024-03-25T18:09:08.793 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.792+0000 7f8275ffb640 1 -- 172.21.15.156:0/4038193556 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052568739 0 0) 0x7f82780af940 con 0x7f82780b4d60 2024-03-25T18:09:08.793 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.792+0000 7f8275ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:09:08.793 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.792+0000 7f8275ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:08.793 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.792+0000 7f8275ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:08.793 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.792+0000 7f8275ffb640 10 monclient(hunting): my global_id is 4257 2024-03-25T18:09:08.793 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.792+0000 7f8275ffb640 1 -- 172.21.15.156:0/4038193556 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82780b0a70 con 0x7f82780b4d60 2024-03-25T18:09:08.801 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.800+0000 7f8275ffb640 1 -- 172.21.15.156:0/4038193556 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 251934330 0 0) 0x7f82780b0a70 con 0x7f82780b4d60 2024-03-25T18:09:08.801 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.800+0000 7f8275ffb640 1 -- 172.21.15.156:0/4038193556 >> v1:172.21.15.156:6790/0 conn(0x7f82780b1730 legacy=0x7f82780b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:08.801 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.800+0000 7f8275ffb640 1 -- 172.21.15.156:0/4038193556 >> v1:172.21.15.194:6789/0 conn(0x7f827814be90 legacy=0x7f827814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:08.801 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.800+0000 7f8275ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:09:08.801 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.800+0000 7f8275ffb640 1 monclient: found mon.noname-a 2024-03-25T18:09:08.801 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.800+0000 7f8275ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:08.801 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.800+0000 7f8275ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:09:08.801 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.800+0000 7f8275ffb640 1 -- 172.21.15.156:0/4038193556 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82780b1030 con 0x7f82780b4d60 2024-03-25T18:09:08.801 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.800+0000 7f8275ffb640 10 monclient: _finish_auth 0 2024-03-25T18:09:08.801 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.800+0000 7f8275ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:09:08.801 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.800+0000 7f8275ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:08.801 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.800+0000 7f8275ffb640 1 -- 172.21.15.156:0/4038193556 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f8268001670 con 0x7f82780b4d60 2024-03-25T18:09:08.802 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.800+0000 7f8275ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:08.802 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.800+0000 7f8275ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:08.802 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.800+0000 7f8275ffb640 10 monclient: dump: 2024-03-25T18:09:08.802 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:08.802 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:08.802 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:08.802 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:08.802 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:08.802 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:08.802 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:08.802 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:08.802 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:08.802 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:08.802 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.800+0000 7f827d94d640 5 monclient: authenticate success, global_id 4257 2024-03-25T18:09:08.802 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.800+0000 7f827d94d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:08.806 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.806+0000 7f8275ffb640 1 -- 172.21.15.156:0/4038193556 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82680017d0 con 0x7f82780b4d60 2024-03-25T18:09:08.807 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.806+0000 7f8275ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:08.807 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.806+0000 7f8275ffb640 1 -- 172.21.15.156:0/4038193556 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f8268001ba0 con 0x7f82780b4d60 2024-03-25T18:09:08.807 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.806+0000 7f8275ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:08.807 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.806+0000 7f8275ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:08.807 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.806+0000 7f8275ffb640 10 monclient: dump: 2024-03-25T18:09:08.807 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:08.807 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:08.807 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:08.807 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:08.807 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:08.807 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:08.807 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:08.807 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:08.807 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:08.807 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:08.808 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.806+0000 7f827d94d640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:08.808 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.806+0000 7f827d94d640 4 set_mon_vals no callback set 2024-03-25T18:09:08.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.807+0000 7f827d94d640 10 monclient: shutdown 2024-03-25T18:09:08.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.807+0000 7f827d94d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:08.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.807+0000 7f827d94d640 1 -- 172.21.15.156:0/4038193556 >> v1:172.21.15.156:6789/0 conn(0x7f82780b4d60 legacy=0x7f827813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:08.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.807+0000 7f827d94d640 1 -- 172.21.15.156:0/4038193556 shutdown_connections 2024-03-25T18:09:08.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.807+0000 7f827d94d640 1 -- 172.21.15.156:0/4038193556 >> 172.21.15.156:0/4038193556 conn(0x7f82780617a0 msgr2=0x7f8278061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:08.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.807+0000 7f827d94d640 1 -- 172.21.15.156:0/4038193556 shutdown_connections 2024-03-25T18:09:08.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.807+0000 7f827d94d640 1 -- 172.21.15.156:0/4038193556 wait complete. 2024-03-25T18:09:08.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.808+0000 7f827d94d640 10 monclient: build_initial_monmap 2024-03-25T18:09:08.810 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.808+0000 7f827d94d640 10 monclient: monmap: 2024-03-25T18:09:08.810 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:09:08.810 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:08.810 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:09:08.810 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:09:08.810 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:08.810 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:09:08.810 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:09:08.810 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:09:08.810 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:09:08.810 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:08.810 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.808+0000 7f827d94d640 1 librados: starting msgr at 2024-03-25T18:09:08.810 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.808+0000 7f827d94d640 1 librados: starting objecter 2024-03-25T18:09:08.810 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.809+0000 7f827d94d640 1 Processor -- start 2024-03-25T18:09:08.811 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.809+0000 7f827d94d640 1 -- start start 2024-03-25T18:09:08.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.809+0000 7f827d94d640 1 librados: setting wanted keys 2024-03-25T18:09:08.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.809+0000 7f827d94d640 1 librados: calling monclient init 2024-03-25T18:09:08.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.809+0000 7f827d94d640 10 monclient: init 2024-03-25T18:09:08.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.809+0000 7f827d94d640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:08.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.809+0000 7f827d94d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:08.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.809+0000 7f827d94d640 10 monclient(hunting): picked mon.noname-b con 0x7f82780b1730 addr v1:172.21.15.194:6789/0 2024-03-25T18:09:08.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.809+0000 7f827d94d640 10 monclient(hunting): picked mon.noname-c con 0x7f82780b4d60 addr v1:172.21.15.156:6790/0 2024-03-25T18:09:08.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.809+0000 7f827d94d640 10 monclient(hunting): picked mon.noname-a con 0x7f827814be90 addr v1:172.21.15.156:6789/0 2024-03-25T18:09:08.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.809+0000 7f827d94d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f827813a960 con 0x7f827814be90 2024-03-25T18:09:08.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.810+0000 7f827d94d640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f827813ac40 con 0x7f82780b1730 2024-03-25T18:09:08.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.810+0000 7f827d94d640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8278139a50 con 0x7f82780b4d60 2024-03-25T18:09:08.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.810+0000 7f827d94d640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:08.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.810+0000 7f827d94d640 10 monclient(hunting): authenticate will time out at 7030.120849s 2024-03-25T18:09:08.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.819+0000 7f82767fc640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f82780b4d60 0x7f8278131da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:44720/0 (socket says 172.21.15.156:44720) 2024-03-25T18:09:08.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.819+0000 7f82767fc640 1 -- 172.21.15.156:0/2936559671 learned_addr learned my addr 172.21.15.156:0/2936559671 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:09:08.824 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.824+0000 7f825ffff640 10 client.?.objecter ms_handle_connect 0x7f82780b4d60 2024-03-25T18:09:08.824 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.824+0000 7f825ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:08.829 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.828+0000 7f825ffff640 10 client.?.objecter ms_handle_connect 0x7f827814be90 2024-03-25T18:09:08.829 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.828+0000 7f825ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:08.830 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.830+0000 7f825ffff640 10 client.?.objecter ms_handle_connect 0x7f82780b1730 2024-03-25T18:09:08.831 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.830+0000 7f825ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:08.833 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.832+0000 7f825ffff640 1 -- 172.21.15.156:0/2936559671 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494737379 0 0) 0x7f827813a960 con 0x7f827814be90 2024-03-25T18:09:08.833 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.832+0000 7f825ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:08.833 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.832+0000 7f825ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:08.833 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.832+0000 7f825ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:08.833 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.832+0000 7f825ffff640 10 monclient(hunting): my global_id is 4260 2024-03-25T18:09:08.833 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.832+0000 7f825ffff640 1 -- 172.21.15.156:0/2936559671 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8264004eb0 con 0x7f827814be90 2024-03-25T18:09:08.834 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.834+0000 7f825ffff640 1 -- 172.21.15.156:0/2936559671 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3485976904 0 0) 0x7f8278139a50 con 0x7f82780b4d60 2024-03-25T18:09:08.835 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.834+0000 7f825ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:08.835 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.834+0000 7f825ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:08.835 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.834+0000 7f825ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:08.835 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.834+0000 7f825ffff640 10 monclient(hunting): my global_id is 4268 2024-03-25T18:09:08.835 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.834+0000 7f825ffff640 1 -- 172.21.15.156:0/2936559671 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f827813a960 con 0x7f82780b4d60 2024-03-25T18:09:08.835 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.834+0000 7f825ffff640 1 -- 172.21.15.156:0/2936559671 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3696958541 0 0) 0x7f827813ac40 con 0x7f82780b1730 2024-03-25T18:09:08.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.834+0000 7f825ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:08.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.834+0000 7f825ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:08.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.834+0000 7f825ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:08.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.834+0000 7f825ffff640 10 monclient(hunting): my global_id is 4258 2024-03-25T18:09:08.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.834+0000 7f825ffff640 1 -- 172.21.15.156:0/2936559671 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8278139a50 con 0x7f82780b1730 2024-03-25T18:09:08.848 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.847+0000 7f825ffff640 1 -- 172.21.15.156:0/2936559671 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 168576847 0 0) 0x7f8278139a50 con 0x7f82780b1730 2024-03-25T18:09:08.848 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.847+0000 7f825ffff640 1 -- 172.21.15.156:0/2936559671 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f827813ac40 con 0x7f82780b1730 2024-03-25T18:09:08.848 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.847+0000 7f825ffff640 1 -- 172.21.15.156:0/2936559671 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f8268001ae0 con 0x7f82780b1730 2024-03-25T18:09:08.848 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.847+0000 7f825ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:08.848 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.847+0000 7f825ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:08.848 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.847+0000 7f825ffff640 10 monclient(hunting): dump: 2024-03-25T18:09:08.848 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:08.848 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:08.848 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:08.849 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:08.849 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:08.849 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:08.849 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:08.849 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:08.849 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:08.849 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:08.849 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.847+0000 7f825ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:08.852 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.851+0000 7f825ffff640 1 -- 172.21.15.156:0/2936559671 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1288439680 0 0) 0x7f8264004eb0 con 0x7f827814be90 2024-03-25T18:09:08.852 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.851+0000 7f825ffff640 1 -- 172.21.15.156:0/2936559671 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8278139a50 con 0x7f827814be90 2024-03-25T18:09:08.853 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.851+0000 7f825ffff640 1 -- 172.21.15.156:0/2936559671 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f8270001b90 con 0x7f827814be90 2024-03-25T18:09:08.853 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.851+0000 7f825ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:08.853 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.852+0000 7f825ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:08.853 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.852+0000 7f825ffff640 10 monclient(hunting): dump: 2024-03-25T18:09:08.853 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:08.853 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:08.853 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:08.853 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:08.853 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:08.853 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:08.853 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:08.853 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:08.853 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:08.853 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:08.853 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.852+0000 7f825ffff640 1 -- 172.21.15.156:0/2936559671 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2007602685 0 0) 0x7f827813a960 con 0x7f82780b4d60 2024-03-25T18:09:08.853 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.852+0000 7f825ffff640 1 -- 172.21.15.156:0/2936559671 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8264004eb0 con 0x7f82780b4d60 2024-03-25T18:09:08.854 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.852+0000 7f825ffff640 1 -- 172.21.15.156:0/2936559671 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f826c001970 con 0x7f82780b4d60 2024-03-25T18:09:08.854 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.852+0000 7f825ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:08.854 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.852+0000 7f825ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:08.854 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.852+0000 7f825ffff640 10 monclient(hunting): dump: 2024-03-25T18:09:08.854 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:08.854 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:08.854 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:08.854 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:08.854 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:08.854 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:08.854 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:08.854 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:08.854 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:08.854 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:08.862 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.861+0000 7f825ffff640 1 -- 172.21.15.156:0/2936559671 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3686067664 0 0) 0x7f8278139a50 con 0x7f827814be90 2024-03-25T18:09:08.862 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.861+0000 7f825ffff640 1 -- 172.21.15.156:0/2936559671 >> v1:172.21.15.156:6790/0 conn(0x7f82780b4d60 legacy=0x7f8278131da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:08.862 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.861+0000 7f825ffff640 1 -- 172.21.15.156:0/2936559671 >> v1:172.21.15.194:6789/0 conn(0x7f82780b1730 legacy=0x7f827813b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:08.862 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.861+0000 7f825ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:09:08.862 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.861+0000 7f825ffff640 1 monclient: found mon.a 2024-03-25T18:09:08.862 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.861+0000 7f825ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:08.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.861+0000 7f825ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:09:08.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.861+0000 7f825ffff640 1 -- 172.21.15.156:0/2936559671 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8278139be0 con 0x7f827814be90 2024-03-25T18:09:08.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.861+0000 7f825ffff640 10 monclient: _finish_auth 0 2024-03-25T18:09:08.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.861+0000 7f825ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:09:08.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.861+0000 7f825ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:08.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.861+0000 7f827d94d640 5 monclient: authenticate success, global_id 4260 2024-03-25T18:09:08.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.861+0000 7f827d94d640 10 monclient: _renew_subs 2024-03-25T18:09:08.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.861+0000 7f827d94d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:09:08.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.861+0000 7f827d94d640 1 -- 172.21.15.156:0/2936559671 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8278138180 con 0x7f827814be90 2024-03-25T18:09:08.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.861+0000 7f827d94d640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:08.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.861+0000 7f827d94d640 10 monclient: _renew_subs 2024-03-25T18:09:08.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.861+0000 7f827d94d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:09:08.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.861+0000 7f827d94d640 1 -- 172.21.15.156:0/2936559671 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8278139040 con 0x7f827814be90 2024-03-25T18:09:08.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.861+0000 7f827d94d640 1 librados: init done 2024-03-25T18:09:08.864 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.862+0000 7f827d94d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:08.864 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.862+0000 7f827d94d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:08.864 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.862+0000 7f827d94d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:09:08.864 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:08.862+0000 7f827d94d640 1 -- 172.21.15.156:0/2936559671 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82780627b0 con 0x7f827814be90 2024-03-25T18:09:09.165 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.164+0000 7f825ffff640 1 -- 172.21.15.156:0/2936559671 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8270001fe0 con 0x7f827814be90 2024-03-25T18:09:09.165 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.164+0000 7f825ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:09.165 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.164+0000 7f825ffff640 1 -- 172.21.15.156:0/2936559671 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f8270004d00 con 0x7f827814be90 2024-03-25T18:09:09.165 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.164+0000 7f825ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:09.165 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.164+0000 7f825ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:09.165 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.164+0000 7f825ffff640 10 monclient: dump: 2024-03-25T18:09:09.165 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:09.165 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:09.166 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:09.166 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:09.166 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:09.166 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:09.166 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:09.166 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:09.166 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:09.166 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:09.166 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.164+0000 7f82757fa640 4 set_mon_vals no callback set 2024-03-25T18:09:09.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.165+0000 7f825ffff640 1 -- 172.21.15.156:0/2936559671 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f8270004f20 con 0x7f827814be90 2024-03-25T18:09:09.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.166+0000 7f8276ffd640 10 monclient: get_auth_request con 0x7f826406e110 auth_method 0 2024-03-25T18:09:09.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.166+0000 7f825ffff640 10 client.4260.objecter ms_handle_connect 0x7f826406e110 2024-03-25T18:09:09.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.171+0000 7f825ffff640 1 -- 172.21.15.156:0/2936559671 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f82700549a0 con 0x7f827814be90 2024-03-25T18:09:09.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.171+0000 7f825ffff640 10 client.4260.objecter ms_dispatch 0x7f8278000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:09.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.171+0000 7f825ffff640 3 client.4260.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:09.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.171+0000 7f825ffff640 3 client.4260.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:09.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.171+0000 7f825ffff640 20 client.4260.objecter dump_active .. 0 homeless 2024-03-25T18:09:09.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.171+0000 7f825ffff640 1 -- 172.21.15.156:0/2936559671 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f82700504b0 con 0x7f827814be90 2024-03-25T18:09:09.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.171+0000 7f825ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:09.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.171+0000 7f825ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:09.427 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.426+0000 7f827d94d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:09.427 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.426+0000 7f827d94d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:09.427 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.426+0000 7f827d94d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:09:09.428 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.426+0000 7f827d94d640 1 -- 172.21.15.156:0/2936559671 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82780b0250 con 0x7f827814be90 2024-03-25T18:09:09.455 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.454+0000 7f825ffff640 1 -- 172.21.15.156:0/2936559671 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f82700523e0 con 0x7f827814be90 2024-03-25T18:09:09.455 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.454+0000 7f825ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:09.455 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.454+0000 7f825ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:09.455 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:09:09.455 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:09.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.457+0000 7f827d94d640 10 librados: watch_flush enter 2024-03-25T18:09:09.459 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.457+0000 7f827d94d640 10 librados: watch_flush exit 2024-03-25T18:09:09.459 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.457+0000 7f827d94d640 20 client.4260.objecter shutdown clearing up homeless session... 2024-03-25T18:09:09.459 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.457+0000 7f827d94d640 10 client.4260.objecter successfully canceled tick 2024-03-25T18:09:09.459 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.457+0000 7f827d94d640 1 -- 172.21.15.156:0/2936559671 >> v1:172.21.15.194:6800/2946427713 conn(0x7f826406e110 legacy=0x7f82640705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:09.459 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.457+0000 7f827d94d640 10 monclient: shutdown 2024-03-25T18:09:09.459 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.457+0000 7f827d94d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:09.459 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.457+0000 7f827d94d640 1 -- 172.21.15.156:0/2936559671 >> v1:172.21.15.156:6789/0 conn(0x7f827814be90 legacy=0x7f82781e3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:09.459 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.457+0000 7f827d94d640 1 -- 172.21.15.156:0/2936559671 shutdown_connections 2024-03-25T18:09:09.459 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.457+0000 7f827d94d640 1 -- 172.21.15.156:0/2936559671 >> 172.21.15.156:0/2936559671 conn(0x7f82780617a0 msgr2=0x7f827813c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:09.459 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.458+0000 7f827d94d640 1 -- 172.21.15.156:0/2936559671 shutdown_connections 2024-03-25T18:09:09.459 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.458+0000 7f827d94d640 1 -- 172.21.15.156:0/2936559671 wait complete. 2024-03-25T18:09:09.459 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:09.458+0000 7f827d94d640 1 librados: shutdown 2024-03-25T18:09:09.472 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:13.073 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:01:19.063 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:13.128 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:01:19.119 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:13.184 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:01:19.151 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:13.244 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:01:19.211 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:13.269 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:01:19.259 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:13.325 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:01:19.316 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:13.349 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:01:19.340 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:13.464 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:01:19.431 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:13.566 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:01:19.557 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:16.277 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:16.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.384+0000 7f4735da8640 10 monclient: get_monmap_and_config 2024-03-25T18:09:16.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.384+0000 7f4735da8640 10 monclient: build_initial_monmap 2024-03-25T18:09:16.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.384+0000 7f4735da8640 10 monclient: monmap: 2024-03-25T18:09:16.385 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:09:16.385 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:16.385 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:09:16.385431+0000 2024-03-25T18:09:16.385 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:09:16.385431+0000 2024-03-25T18:09:16.385 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:16.385 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:16.385 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:09:16.385 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:09:16.386 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:09:16.386 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:16.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.385+0000 7f4735da8640 1 Processor -- start 2024-03-25T18:09:16.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.385+0000 7f4735da8640 1 -- start start 2024-03-25T18:09:16.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.385+0000 7f4735da8640 10 monclient: init 2024-03-25T18:09:16.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.385+0000 7f4735da8640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:16.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.385+0000 7f4735da8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:16.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.385+0000 7f4735da8640 10 monclient(hunting): picked mon.noname-c con 0x7f47300a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:09:16.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.385+0000 7f4735da8640 10 monclient(hunting): picked mon.noname-b con 0x7f473013f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:09:16.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.385+0000 7f4735da8640 10 monclient(hunting): picked mon.noname-a con 0x7f47301430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:09:16.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.385+0000 7f4735da8640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4730142520 con 0x7f47301430c0 2024-03-25T18:09:16.388 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.385+0000 7f4735da8640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47300b28a0 con 0x7f47300a98f0 2024-03-25T18:09:16.388 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.385+0000 7f4735da8640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47300b2a30 con 0x7f473013f710 2024-03-25T18:09:16.388 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.385+0000 7f4735da8640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:16.388 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.385+0000 7f4735da8640 10 monclient(hunting): authenticate will time out at 7037.696733s 2024-03-25T18:09:16.388 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.387+0000 7f472effd640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f473013f710 0x7f47301429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:57178/0 (socket says 172.21.15.156:57178) 2024-03-25T18:09:16.388 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.387+0000 7f472effd640 1 -- 172.21.15.156:0/3211183579 learned_addr learned my addr 172.21.15.156:0/3211183579 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:09:16.447 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.446+0000 7f472e7fc640 1 -- 172.21.15.156:0/3211183579 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733385064 0 0) 0x7f47300b28a0 con 0x7f47300a98f0 2024-03-25T18:09:16.447 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.446+0000 7f472e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:16.447 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.446+0000 7f472e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:16.447 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.446+0000 7f472e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:16.447 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.446+0000 7f472e7fc640 10 monclient(hunting): my global_id is 4261 2024-03-25T18:09:16.447 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.446+0000 7f472e7fc640 1 -- 172.21.15.156:0/3211183579 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4718003520 con 0x7f47300a98f0 2024-03-25T18:09:16.467 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.466+0000 7f472e7fc640 1 -- 172.21.15.156:0/3211183579 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149909278 0 0) 0x7f47300b2a30 con 0x7f473013f710 2024-03-25T18:09:16.467 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.466+0000 7f472e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:16.467 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.466+0000 7f472e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:16.467 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.466+0000 7f472e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:16.467 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.466+0000 7f472e7fc640 10 monclient(hunting): my global_id is 4271 2024-03-25T18:09:16.467 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.466+0000 7f472e7fc640 1 -- 172.21.15.156:0/3211183579 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47300b28a0 con 0x7f473013f710 2024-03-25T18:09:16.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.468+0000 7f472e7fc640 1 -- 172.21.15.156:0/3211183579 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3073045511 0 0) 0x7f4718003520 con 0x7f47300a98f0 2024-03-25T18:09:16.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.468+0000 7f472e7fc640 1 -- 172.21.15.156:0/3211183579 >> v1:172.21.15.156:6790/0 conn(0x7f473013f710 legacy=0x7f47301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:16.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.468+0000 7f472e7fc640 1 -- 172.21.15.156:0/3211183579 >> v1:172.21.15.156:6789/0 conn(0x7f47301430c0 legacy=0x7f473014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:16.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.468+0000 7f472e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:09:16.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.468+0000 7f472e7fc640 1 monclient: found mon.noname-c 2024-03-25T18:09:16.470 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.468+0000 7f472e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:16.470 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.468+0000 7f472e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:09:16.470 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.468+0000 7f472e7fc640 1 -- 172.21.15.156:0/3211183579 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47300b44c0 con 0x7f47300a98f0 2024-03-25T18:09:16.470 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.468+0000 7f472e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:09:16.470 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.468+0000 7f472e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:09:16.470 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.468+0000 7f472e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:16.470 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.468+0000 7f472e7fc640 1 -- 172.21.15.156:0/3211183579 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f4720001690 con 0x7f47300a98f0 2024-03-25T18:09:16.470 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.468+0000 7f472e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:16.470 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.468+0000 7f472e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:16.470 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.468+0000 7f472e7fc640 10 monclient: dump: 2024-03-25T18:09:16.470 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:16.470 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:16.470 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:16.470 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:16.471 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:16.471 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:16.471 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:16.471 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:16.471 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:16.471 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:16.471 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.468+0000 7f4735da8640 5 monclient: authenticate success, global_id 4261 2024-03-25T18:09:16.471 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.468+0000 7f4735da8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:16.480 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.480+0000 7f472e7fc640 1 -- 172.21.15.156:0/3211183579 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4720001870 con 0x7f47300a98f0 2024-03-25T18:09:16.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.480+0000 7f472e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:16.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.480+0000 7f472e7fc640 1 -- 172.21.15.156:0/3211183579 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f4720001c80 con 0x7f47300a98f0 2024-03-25T18:09:16.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.480+0000 7f4735da8640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:16.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.480+0000 7f4735da8640 4 set_mon_vals no callback set 2024-03-25T18:09:16.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.480+0000 7f472e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:16.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.480+0000 7f472e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:16.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.480+0000 7f472e7fc640 10 monclient: dump: 2024-03-25T18:09:16.481 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:16.481 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:16.481 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:16.481 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:16.481 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:16.481 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:16.481 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:16.482 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:16.482 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:16.482 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:16.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.481+0000 7f4735da8640 10 monclient: shutdown 2024-03-25T18:09:16.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.481+0000 7f4735da8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:16.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.481+0000 7f4735da8640 1 -- 172.21.15.156:0/3211183579 >> v1:172.21.15.194:6789/0 conn(0x7f47300a98f0 legacy=0x7f47300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:16.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.481+0000 7f4735da8640 1 -- 172.21.15.156:0/3211183579 shutdown_connections 2024-03-25T18:09:16.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.481+0000 7f4735da8640 1 -- 172.21.15.156:0/3211183579 >> 172.21.15.156:0/3211183579 conn(0x7f47300b7ea0 msgr2=0x7f47300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:16.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.481+0000 7f4735da8640 1 -- 172.21.15.156:0/3211183579 shutdown_connections 2024-03-25T18:09:16.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.481+0000 7f4735da8640 1 -- 172.21.15.156:0/3211183579 wait complete. 2024-03-25T18:09:16.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.482+0000 7f4735da8640 10 monclient: build_initial_monmap 2024-03-25T18:09:16.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.482+0000 7f4735da8640 10 monclient: monmap: 2024-03-25T18:09:16.483 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:09:16.484 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:16.484 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:09:16.484 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:09:16.484 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:16.484 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:09:16.484 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:09:16.484 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:09:16.484 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:09:16.484 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:16.484 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.482+0000 7f4735da8640 1 librados: starting msgr at 2024-03-25T18:09:16.484 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.482+0000 7f4735da8640 1 librados: starting objecter 2024-03-25T18:09:16.485 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.482+0000 7f4735da8640 1 Processor -- start 2024-03-25T18:09:16.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.483+0000 7f4735da8640 1 -- start start 2024-03-25T18:09:16.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.483+0000 7f4735da8640 1 librados: setting wanted keys 2024-03-25T18:09:16.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.483+0000 7f4735da8640 1 librados: calling monclient init 2024-03-25T18:09:16.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.483+0000 7f4735da8640 10 monclient: init 2024-03-25T18:09:16.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.483+0000 7f4735da8640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:16.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.483+0000 7f4735da8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:09:16.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.483+0000 7f4735da8640 10 monclient(hunting): picked mon.noname-c con 0x7f47300a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:09:16.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.483+0000 7f4735da8640 10 monclient(hunting): picked mon.noname-a con 0x7f473013f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:09:16.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.484+0000 7f4735da8640 10 monclient(hunting): picked mon.noname-b con 0x7f47301430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:09:16.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.484+0000 7f4735da8640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4730135730 con 0x7f473013f710 2024-03-25T18:09:16.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.484+0000 7f4735da8640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47301358c0 con 0x7f47301430c0 2024-03-25T18:09:16.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.484+0000 7f4735da8640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4730135a50 con 0x7f47300a98f0 2024-03-25T18:09:16.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.484+0000 7f4735da8640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:16.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.484+0000 7f4735da8640 10 monclient(hunting): authenticate will time out at 7037.794964s 2024-03-25T18:09:16.488 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.488+0000 7f472effd640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f473013f710 0x7f47301d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:52838/0 (socket says 172.21.15.156:52838) 2024-03-25T18:09:16.488 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.488+0000 7f472effd640 1 -- 172.21.15.156:0/1177803912 learned_addr learned my addr 172.21.15.156:0/1177803912 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:09:16.489 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.488+0000 7f4717fff640 10 client.?.objecter ms_handle_connect 0x7f473013f710 2024-03-25T18:09:16.489 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.489+0000 7f4717fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:16.493 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.492+0000 7f4717fff640 1 -- 172.21.15.156:0/1177803912 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4145906361 0 0) 0x7f4730135730 con 0x7f473013f710 2024-03-25T18:09:16.493 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.492+0000 7f4717fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:16.493 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.492+0000 7f4717fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:16.493 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.492+0000 7f4717fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:16.493 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.492+0000 7f4717fff640 10 monclient(hunting): my global_id is 4266 2024-03-25T18:09:16.493 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.492+0000 7f4717fff640 1 -- 172.21.15.156:0/1177803912 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f471c004eb0 con 0x7f473013f710 2024-03-25T18:09:16.498 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.497+0000 7f4717fff640 10 client.?.objecter ms_handle_connect 0x7f47300a98f0 2024-03-25T18:09:16.498 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.497+0000 7f4717fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:16.501 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.500+0000 7f4717fff640 10 client.?.objecter ms_handle_connect 0x7f47301430c0 2024-03-25T18:09:16.501 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.500+0000 7f4717fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:16.506 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.503+0000 7f4717fff640 1 -- 172.21.15.156:0/1177803912 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1603595123 0 0) 0x7f471c004eb0 con 0x7f473013f710 2024-03-25T18:09:16.506 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.503+0000 7f4717fff640 1 -- 172.21.15.156:0/1177803912 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4730135730 con 0x7f473013f710 2024-03-25T18:09:16.506 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.503+0000 7f4717fff640 1 -- 172.21.15.156:0/1177803912 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f47240019b0 con 0x7f473013f710 2024-03-25T18:09:16.506 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.503+0000 7f4717fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:16.506 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.503+0000 7f4717fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:16.506 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.503+0000 7f4717fff640 10 monclient(hunting): dump: 2024-03-25T18:09:16.506 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:16.506 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:16.506 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:16.507 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:16.507 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:16.507 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:16.507 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:16.507 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:16.507 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:16.507 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:16.507 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.503+0000 7f4717fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:16.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.507+0000 7f4717fff640 1 -- 172.21.15.156:0/1177803912 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3172117455 0 0) 0x7f4730135a50 con 0x7f47300a98f0 2024-03-25T18:09:16.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.507+0000 7f4717fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:16.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.507+0000 7f4717fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:16.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.507+0000 7f4717fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:16.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.507+0000 7f4717fff640 10 monclient(hunting): my global_id is 4274 2024-03-25T18:09:16.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.507+0000 7f4717fff640 1 -- 172.21.15.156:0/1177803912 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f471c004eb0 con 0x7f47300a98f0 2024-03-25T18:09:16.514 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.513+0000 7f4717fff640 1 -- 172.21.15.156:0/1177803912 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1313237834 0 0) 0x7f4730135730 con 0x7f473013f710 2024-03-25T18:09:16.514 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.513+0000 7f4717fff640 1 -- 172.21.15.156:0/1177803912 >> v1:172.21.15.156:6790/0 conn(0x7f47300a98f0 legacy=0x7f47301dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:16.514 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.513+0000 7f4717fff640 1 -- 172.21.15.156:0/1177803912 >> v1:172.21.15.194:6789/0 conn(0x7f47301430c0 legacy=0x7f47301d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:16.514 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.513+0000 7f4717fff640 10 monclient: _finish_hunting 0 2024-03-25T18:09:16.514 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.513+0000 7f4717fff640 1 monclient: found mon.a 2024-03-25T18:09:16.514 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.513+0000 7f4717fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:16.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.513+0000 7f4717fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:09:16.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.513+0000 7f4717fff640 1 -- 172.21.15.156:0/1177803912 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47301d5000 con 0x7f473013f710 2024-03-25T18:09:16.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.513+0000 7f4717fff640 10 monclient: _finish_auth 0 2024-03-25T18:09:16.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.513+0000 7f4717fff640 10 monclient: _check_auth_tickets 2024-03-25T18:09:16.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.513+0000 7f4717fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:16.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.513+0000 7f4735da8640 5 monclient: authenticate success, global_id 4266 2024-03-25T18:09:16.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.513+0000 7f4735da8640 10 monclient: _renew_subs 2024-03-25T18:09:16.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.513+0000 7f4735da8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:09:16.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.513+0000 7f4735da8640 1 -- 172.21.15.156:0/1177803912 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47301d52b0 con 0x7f473013f710 2024-03-25T18:09:16.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.513+0000 7f4735da8640 10 client.4266.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:16.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.513+0000 7f4735da8640 10 monclient: _renew_subs 2024-03-25T18:09:16.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.513+0000 7f4735da8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:09:16.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.513+0000 7f4735da8640 1 -- 172.21.15.156:0/1177803912 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f473013c900 con 0x7f473013f710 2024-03-25T18:09:16.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.513+0000 7f4735da8640 1 librados: init done 2024-03-25T18:09:16.516 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.514+0000 7f4735da8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:16.516 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.514+0000 7f4735da8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:16.516 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.514+0000 7f4735da8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:09:16.516 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.514+0000 7f4735da8640 1 -- 172.21.15.156:0/1177803912 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47301371c0 con 0x7f473013f710 2024-03-25T18:09:16.838 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.837+0000 7f4717fff640 1 -- 172.21.15.156:0/1177803912 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47240047e0 con 0x7f473013f710 2024-03-25T18:09:16.848 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.837+0000 7f4717fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:16.849 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.837+0000 7f4717fff640 1 -- 172.21.15.156:0/1177803912 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f4724004bf0 con 0x7f473013f710 2024-03-25T18:09:16.849 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.837+0000 7f4717fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:16.849 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.837+0000 7f4717fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:16.849 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.837+0000 7f4717fff640 10 monclient: dump: 2024-03-25T18:09:16.849 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:16.849 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:16.849 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:16.849 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:16.849 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:16.849 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:16.849 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:16.849 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:16.849 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:16.850 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:16.850 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.837+0000 7f472dffb640 4 set_mon_vals no callback set 2024-03-25T18:09:16.851 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.838+0000 7f4717fff640 1 -- 172.21.15.156:0/1177803912 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f4724004e10 con 0x7f473013f710 2024-03-25T18:09:16.851 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.838+0000 7f472f7fe640 10 monclient: get_auth_request con 0x7f471c06dd90 auth_method 0 2024-03-25T18:09:16.851 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.839+0000 7f4717fff640 10 client.4266.objecter ms_handle_connect 0x7f471c06dd90 2024-03-25T18:09:16.851 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.846+0000 7f4717fff640 1 -- 172.21.15.156:0/1177803912 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f4724054840 con 0x7f473013f710 2024-03-25T18:09:16.851 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.846+0000 7f4717fff640 10 client.4266.objecter ms_dispatch 0x7f4730000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:16.851 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.846+0000 7f4717fff640 3 client.4266.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:16.852 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.846+0000 7f4717fff640 3 client.4266.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:16.852 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.847+0000 7f4717fff640 20 client.4266.objecter dump_active .. 0 homeless 2024-03-25T18:09:16.852 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.847+0000 7f4717fff640 1 -- 172.21.15.156:0/1177803912 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4724050350 con 0x7f473013f710 2024-03-25T18:09:16.852 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.847+0000 7f4717fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:16.852 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:16.847+0000 7f4717fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:17.106 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:17.105+0000 7f4735da8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:17.106 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:17.105+0000 7f4735da8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:17.107 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:17.105+0000 7f4735da8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:09:17.107 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:17.105+0000 7f4735da8640 1 -- 172.21.15.156:0/1177803912 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47300627b0 con 0x7f473013f710 2024-03-25T18:09:17.130 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:17.130+0000 7f4717fff640 1 -- 172.21.15.156:0/1177803912 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f4724052280 con 0x7f473013f710 2024-03-25T18:09:17.131 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:17.130+0000 7f4717fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:17.131 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:17.130+0000 7f4717fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:17.131 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:09:17.131 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:17.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:17.132+0000 7f4735da8640 10 librados: watch_flush enter 2024-03-25T18:09:17.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:17.132+0000 7f4735da8640 10 librados: watch_flush exit 2024-03-25T18:09:17.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:17.132+0000 7f4735da8640 20 client.4266.objecter shutdown clearing up homeless session... 2024-03-25T18:09:17.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:17.132+0000 7f4735da8640 10 client.4266.objecter successfully canceled tick 2024-03-25T18:09:17.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:17.132+0000 7f4735da8640 1 -- 172.21.15.156:0/1177803912 >> v1:172.21.15.194:6800/2946427713 conn(0x7f471c06dd90 legacy=0x7f471c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:17.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:17.132+0000 7f4735da8640 10 monclient: shutdown 2024-03-25T18:09:17.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:17.132+0000 7f4735da8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:17.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:17.132+0000 7f4735da8640 1 -- 172.21.15.156:0/1177803912 >> v1:172.21.15.156:6789/0 conn(0x7f473013f710 legacy=0x7f47301d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:17.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:17.132+0000 7f4735da8640 1 -- 172.21.15.156:0/1177803912 shutdown_connections 2024-03-25T18:09:17.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:17.132+0000 7f4735da8640 1 -- 172.21.15.156:0/1177803912 >> 172.21.15.156:0/1177803912 conn(0x7f47300b7ea0 msgr2=0x7f473014ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:17.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:17.132+0000 7f4735da8640 1 -- 172.21.15.156:0/1177803912 shutdown_connections 2024-03-25T18:09:17.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:17.133+0000 7f4735da8640 1 -- 172.21.15.156:0/1177803912 wait complete. 2024-03-25T18:09:17.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:17.133+0000 7f4735da8640 1 librados: shutdown 2024-03-25T18:09:17.146 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:23.952 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:24.066 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.065+0000 7f77fb6f6640 10 monclient: get_monmap_and_config 2024-03-25T18:09:24.066 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.065+0000 7f77fb6f6640 10 monclient: build_initial_monmap 2024-03-25T18:09:24.066 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.065+0000 7f77fb6f6640 10 monclient: monmap: 2024-03-25T18:09:24.066 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:09:24.066 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:24.066 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:09:24.066796+0000 2024-03-25T18:09:24.066 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:09:24.066796+0000 2024-03-25T18:09:24.066 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:24.067 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:24.067 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:09:24.067 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:09:24.067 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:09:24.067 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:24.068 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.066+0000 7f77fb6f6640 1 Processor -- start 2024-03-25T18:09:24.068 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.066+0000 7f77fb6f6640 1 -- start start 2024-03-25T18:09:24.068 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.066+0000 7f77fb6f6640 10 monclient: init 2024-03-25T18:09:24.068 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.067+0000 7f77fb6f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:24.068 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.067+0000 7f77fb6f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:09:24.068 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.067+0000 7f77fb6f6640 10 monclient(hunting): picked mon.noname-a con 0x7f77f40b4d60 addr v1:172.21.15.156:6789/0 2024-03-25T18:09:24.068 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.067+0000 7f77fb6f6640 10 monclient(hunting): picked mon.noname-c con 0x7f77f40b1730 addr v1:172.21.15.194:6789/0 2024-03-25T18:09:24.069 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.067+0000 7f77fb6f6640 10 monclient(hunting): picked mon.noname-b con 0x7f77f414be90 addr v1:172.21.15.156:6790/0 2024-03-25T18:09:24.069 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.067+0000 7f77fb6f6640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77f40af940 con 0x7f77f40b4d60 2024-03-25T18:09:24.069 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.067+0000 7f77fb6f6640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77f40b0a70 con 0x7f77f40b1730 2024-03-25T18:09:24.069 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.067+0000 7f77fb6f6640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77f40b0d50 con 0x7f77f414be90 2024-03-25T18:09:24.069 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.067+0000 7f77fb6f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:24.069 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.067+0000 7f77fb6f6640 10 monclient(hunting): authenticate will time out at 7045.378273s 2024-03-25T18:09:24.075 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.074+0000 7f77f9c93640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f77f414be90 0x7f77f414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:55448/0 (socket says 172.21.15.156:55448) 2024-03-25T18:09:24.075 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.074+0000 7f77f9492640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f77f40b4d60 0x7f77f413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:45876/0 (socket says 172.21.15.156:45876) 2024-03-25T18:09:24.075 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.074+0000 7f77f9c93640 1 -- 172.21.15.156:0/3123422051 learned_addr learned my addr 172.21.15.156:0/3123422051 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:09:24.086 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.085+0000 7f77ebfff640 1 -- 172.21.15.156:0/3123422051 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237537760 0 0) 0x7f77f40af940 con 0x7f77f40b4d60 2024-03-25T18:09:24.086 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.085+0000 7f77ebfff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:24.086 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.085+0000 7f77ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:24.086 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.085+0000 7f77ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:24.086 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.085+0000 7f77ebfff640 10 monclient(hunting): my global_id is 4269 2024-03-25T18:09:24.086 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.085+0000 7f77ebfff640 1 -- 172.21.15.156:0/3123422051 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77e0003520 con 0x7f77f40b4d60 2024-03-25T18:09:24.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.087+0000 7f77ebfff640 1 -- 172.21.15.156:0/3123422051 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982094391 0 0) 0x7f77f40b0d50 con 0x7f77f414be90 2024-03-25T18:09:24.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.087+0000 7f77ebfff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:24.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.087+0000 7f77ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:24.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.087+0000 7f77ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:24.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.087+0000 7f77ebfff640 10 monclient(hunting): my global_id is 4277 2024-03-25T18:09:24.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.087+0000 7f77ebfff640 1 -- 172.21.15.156:0/3123422051 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77f40af940 con 0x7f77f414be90 2024-03-25T18:09:24.092 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.091+0000 7f77ebfff640 1 -- 172.21.15.156:0/3123422051 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3043622543 0 0) 0x7f77e0003520 con 0x7f77f40b4d60 2024-03-25T18:09:24.092 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.091+0000 7f77ebfff640 1 -- 172.21.15.156:0/3123422051 >> v1:172.21.15.156:6790/0 conn(0x7f77f414be90 legacy=0x7f77f414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:24.092 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.091+0000 7f77ebfff640 1 -- 172.21.15.156:0/3123422051 >> v1:172.21.15.194:6789/0 conn(0x7f77f40b1730 legacy=0x7f77f40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:24.092 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.091+0000 7f77ebfff640 10 monclient: _finish_hunting 0 2024-03-25T18:09:24.092 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.091+0000 7f77ebfff640 1 monclient: found mon.noname-a 2024-03-25T18:09:24.092 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.091+0000 7f77ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:24.092 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.091+0000 7f77ebfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:09:24.092 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.091+0000 7f77ebfff640 1 -- 172.21.15.156:0/3123422051 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f40b1030 con 0x7f77f40b4d60 2024-03-25T18:09:24.092 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.091+0000 7f77ebfff640 10 monclient: _finish_auth 0 2024-03-25T18:09:24.092 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.091+0000 7f77ebfff640 10 monclient: _check_auth_tickets 2024-03-25T18:09:24.092 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.091+0000 7f77ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:24.092 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.091+0000 7f77ebfff640 1 -- 172.21.15.156:0/3123422051 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f77ec001a10 con 0x7f77f40b4d60 2024-03-25T18:09:24.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.091+0000 7f77fb6f6640 5 monclient: authenticate success, global_id 4269 2024-03-25T18:09:24.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.091+0000 7f77fb6f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:24.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.091+0000 7f77ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:24.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.091+0000 7f77ebfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:24.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.091+0000 7f77ebfff640 10 monclient: dump: 2024-03-25T18:09:24.093 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:24.093 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:24.093 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:24.093 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:24.093 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:24.093 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:24.093 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:24.093 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:24.093 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:24.093 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:24.094 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.091+0000 7f77fb6f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:24.140 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.139+0000 7f77ebfff640 1 -- 172.21.15.156:0/3123422051 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77ec001b70 con 0x7f77f40b4d60 2024-03-25T18:09:24.140 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.139+0000 7f77ebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:24.140 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.139+0000 7f77ebfff640 1 -- 172.21.15.156:0/3123422051 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f77ec001f80 con 0x7f77f40b4d60 2024-03-25T18:09:24.140 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.139+0000 7f77fb6f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:24.140 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.139+0000 7f77fb6f6640 4 set_mon_vals no callback set 2024-03-25T18:09:24.140 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.139+0000 7f77ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:24.140 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.139+0000 7f77ebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:24.140 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.139+0000 7f77ebfff640 10 monclient: dump: 2024-03-25T18:09:24.140 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:24.140 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:24.141 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:24.141 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:24.141 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:24.141 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:24.141 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:24.141 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:24.141 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:24.141 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:24.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.140+0000 7f77fb6f6640 10 monclient: shutdown 2024-03-25T18:09:24.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.140+0000 7f77fb6f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:24.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.140+0000 7f77fb6f6640 1 -- 172.21.15.156:0/3123422051 >> v1:172.21.15.156:6789/0 conn(0x7f77f40b4d60 legacy=0x7f77f413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:24.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.140+0000 7f77fb6f6640 1 -- 172.21.15.156:0/3123422051 shutdown_connections 2024-03-25T18:09:24.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.140+0000 7f77fb6f6640 1 -- 172.21.15.156:0/3123422051 >> 172.21.15.156:0/3123422051 conn(0x7f77f40617a0 msgr2=0x7f77f4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:24.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.140+0000 7f77fb6f6640 1 -- 172.21.15.156:0/3123422051 shutdown_connections 2024-03-25T18:09:24.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.140+0000 7f77fb6f6640 1 -- 172.21.15.156:0/3123422051 wait complete. 2024-03-25T18:09:24.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.141+0000 7f77fb6f6640 10 monclient: build_initial_monmap 2024-03-25T18:09:24.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.141+0000 7f77fb6f6640 10 monclient: monmap: 2024-03-25T18:09:24.143 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:09:24.143 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:24.144 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:09:24.144 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:09:24.144 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:24.144 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:09:24.144 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:09:24.144 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:09:24.144 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:09:24.144 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:24.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.141+0000 7f77fb6f6640 1 librados: starting msgr at 2024-03-25T18:09:24.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.141+0000 7f77fb6f6640 1 librados: starting objecter 2024-03-25T18:09:24.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.141+0000 7f77fb6f6640 1 Processor -- start 2024-03-25T18:09:24.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.141+0000 7f77fb6f6640 1 -- start start 2024-03-25T18:09:24.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.141+0000 7f77fb6f6640 1 librados: setting wanted keys 2024-03-25T18:09:24.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.141+0000 7f77fb6f6640 1 librados: calling monclient init 2024-03-25T18:09:24.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.141+0000 7f77fb6f6640 10 monclient: init 2024-03-25T18:09:24.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.142+0000 7f77fb6f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:24.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.142+0000 7f77fb6f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:24.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.142+0000 7f77fb6f6640 10 monclient(hunting): picked mon.noname-b con 0x7f77f40b1730 addr v1:172.21.15.194:6789/0 2024-03-25T18:09:24.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.142+0000 7f77fb6f6640 10 monclient(hunting): picked mon.noname-c con 0x7f77f40b4d60 addr v1:172.21.15.156:6790/0 2024-03-25T18:09:24.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.142+0000 7f77fb6f6640 10 monclient(hunting): picked mon.noname-a con 0x7f77f414be90 addr v1:172.21.15.156:6789/0 2024-03-25T18:09:24.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.142+0000 7f77fb6f6640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77f4136130 con 0x7f77f414be90 2024-03-25T18:09:24.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.142+0000 7f77fb6f6640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77f4145800 con 0x7f77f40b1730 2024-03-25T18:09:24.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.142+0000 7f77fb6f6640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77f41452b0 con 0x7f77f40b4d60 2024-03-25T18:09:24.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.142+0000 7f77fb6f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:24.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.142+0000 7f77fb6f6640 10 monclient(hunting): authenticate will time out at 7045.453248s 2024-03-25T18:09:24.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.152+0000 7f77f8c91640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f77f40b4d60 0x7f77f41db3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:55498/0 (socket says 172.21.15.156:55498) 2024-03-25T18:09:24.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.152+0000 7f77f8c91640 1 -- 172.21.15.156:0/4048582119 learned_addr learned my addr 172.21.15.156:0/4048582119 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:09:24.154 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.154+0000 7f77e9ffb640 10 client.?.objecter ms_handle_connect 0x7f77f40b4d60 2024-03-25T18:09:24.154 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.154+0000 7f77e9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:24.157 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.157+0000 7f77e9ffb640 10 client.?.objecter ms_handle_connect 0x7f77f40b1730 2024-03-25T18:09:24.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.157+0000 7f77e9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:24.160 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.160+0000 7f77e9ffb640 10 client.?.objecter ms_handle_connect 0x7f77f414be90 2024-03-25T18:09:24.160 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.160+0000 7f77e9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:24.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.168+0000 7f77e9ffb640 1 -- 172.21.15.156:0/4048582119 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1376409132 0 0) 0x7f77f41452b0 con 0x7f77f40b4d60 2024-03-25T18:09:24.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.168+0000 7f77e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:09:24.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.168+0000 7f77e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:24.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.168+0000 7f77e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:24.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.168+0000 7f77e9ffb640 10 monclient(hunting): my global_id is 4280 2024-03-25T18:09:24.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.168+0000 7f77e9ffb640 1 -- 172.21.15.156:0/4048582119 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77d8004eb0 con 0x7f77f40b4d60 2024-03-25T18:09:24.171 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.171+0000 7f77e9ffb640 1 -- 172.21.15.156:0/4048582119 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 549165384 0 0) 0x7f77f4145800 con 0x7f77f40b1730 2024-03-25T18:09:24.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.171+0000 7f77e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:09:24.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.171+0000 7f77e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:24.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.171+0000 7f77e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:24.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.171+0000 7f77e9ffb640 10 monclient(hunting): my global_id is 4270 2024-03-25T18:09:24.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.171+0000 7f77e9ffb640 1 -- 172.21.15.156:0/4048582119 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77f41452b0 con 0x7f77f40b1730 2024-03-25T18:09:24.188 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.188+0000 7f77e9ffb640 1 -- 172.21.15.156:0/4048582119 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1978915977 0 0) 0x7f77f4136130 con 0x7f77f414be90 2024-03-25T18:09:24.188 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.188+0000 7f77e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:09:24.188 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.188+0000 7f77e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:24.188 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.188+0000 7f77e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:24.188 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.188+0000 7f77e9ffb640 10 monclient(hunting): my global_id is 4272 2024-03-25T18:09:24.189 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.188+0000 7f77e9ffb640 1 -- 172.21.15.156:0/4048582119 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77f4145800 con 0x7f77f414be90 2024-03-25T18:09:24.192 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.191+0000 7f77e9ffb640 1 -- 172.21.15.156:0/4048582119 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 211242396 0 0) 0x7f77f41452b0 con 0x7f77f40b1730 2024-03-25T18:09:24.192 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.191+0000 7f77e9ffb640 1 -- 172.21.15.156:0/4048582119 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77f4136130 con 0x7f77f40b1730 2024-03-25T18:09:24.192 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.191+0000 7f77e9ffb640 1 -- 172.21.15.156:0/4048582119 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f77ec001e80 con 0x7f77f40b1730 2024-03-25T18:09:24.192 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.191+0000 7f77e9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:24.192 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.191+0000 7f77e9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:24.192 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.191+0000 7f77e9ffb640 10 monclient(hunting): dump: 2024-03-25T18:09:24.192 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:24.192 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:24.192 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:24.192 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:24.192 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:24.192 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:24.192 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:24.192 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:24.193 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:24.193 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:24.193 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.191+0000 7f77e9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:24.199 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.198+0000 7f77e9ffb640 1 -- 172.21.15.156:0/4048582119 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2803357590 0 0) 0x7f77d8004eb0 con 0x7f77f40b4d60 2024-03-25T18:09:24.199 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.198+0000 7f77e9ffb640 1 -- 172.21.15.156:0/4048582119 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77f41452b0 con 0x7f77f40b4d60 2024-03-25T18:09:24.199 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.198+0000 7f77e9ffb640 1 -- 172.21.15.156:0/4048582119 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f77f0001a30 con 0x7f77f40b4d60 2024-03-25T18:09:24.199 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.198+0000 7f77e9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:24.199 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.198+0000 7f77e9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:24.199 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.198+0000 7f77e9ffb640 10 monclient(hunting): dump: 2024-03-25T18:09:24.199 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:24.199 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:24.199 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:24.199 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:24.200 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:24.200 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:24.200 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:24.200 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:24.200 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:24.200 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:24.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.203+0000 7f77e9ffb640 1 -- 172.21.15.156:0/4048582119 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 839046120 0 0) 0x7f77f4145800 con 0x7f77f414be90 2024-03-25T18:09:24.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.203+0000 7f77e9ffb640 1 -- 172.21.15.156:0/4048582119 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77d8004eb0 con 0x7f77f414be90 2024-03-25T18:09:24.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.203+0000 7f77e9ffb640 1 -- 172.21.15.156:0/4048582119 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f77e4001810 con 0x7f77f414be90 2024-03-25T18:09:24.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.203+0000 7f77e9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:24.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.203+0000 7f77e9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:24.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.203+0000 7f77e9ffb640 10 monclient(hunting): dump: 2024-03-25T18:09:24.204 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:24.204 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:24.204 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:24.204 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:24.204 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:24.204 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:24.204 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:24.204 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:24.204 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:24.204 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:24.211 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.210+0000 7f77e9ffb640 1 -- 172.21.15.156:0/4048582119 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 400233268 0 0) 0x7f77f4136130 con 0x7f77f40b1730 2024-03-25T18:09:24.211 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.210+0000 7f77e9ffb640 1 -- 172.21.15.156:0/4048582119 >> v1:172.21.15.156:6790/0 conn(0x7f77f40b4d60 legacy=0x7f77f41db3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:24.211 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.210+0000 7f77e9ffb640 1 -- 172.21.15.156:0/4048582119 >> v1:172.21.15.156:6789/0 conn(0x7f77f414be90 legacy=0x7f77f41e2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:24.211 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.210+0000 7f77e9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:09:24.211 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.210+0000 7f77e9ffb640 1 monclient: found mon.b 2024-03-25T18:09:24.211 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.210+0000 7f77e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:24.212 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.210+0000 7f77e9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:09:24.212 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.210+0000 7f77e9ffb640 1 -- 172.21.15.156:0/4048582119 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f4145440 con 0x7f77f40b1730 2024-03-25T18:09:24.212 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.210+0000 7f77e9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:09:24.212 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.210+0000 7f77e9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:09:24.212 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.210+0000 7f77e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:24.212 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.210+0000 7f77fb6f6640 5 monclient: authenticate success, global_id 4270 2024-03-25T18:09:24.212 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.210+0000 7f77fb6f6640 10 monclient: _renew_subs 2024-03-25T18:09:24.212 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.210+0000 7f77fb6f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:09:24.212 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.210+0000 7f77fb6f6640 1 -- 172.21.15.156:0/4048582119 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77f4135c10 con 0x7f77f40b1730 2024-03-25T18:09:24.212 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.210+0000 7f77fb6f6640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:24.212 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.210+0000 7f77fb6f6640 10 monclient: _renew_subs 2024-03-25T18:09:24.212 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.210+0000 7f77fb6f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:09:24.212 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.210+0000 7f77fb6f6640 1 -- 172.21.15.156:0/4048582119 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77f413e3f0 con 0x7f77f40b1730 2024-03-25T18:09:24.212 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.210+0000 7f77fb6f6640 1 librados: init done 2024-03-25T18:09:24.213 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.212+0000 7f77fb6f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:24.213 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.212+0000 7f77fb6f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:24.213 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.212+0000 7f77fb6f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:09:24.213 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.212+0000 7f77fb6f6640 1 -- 172.21.15.156:0/4048582119 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77f41362c0 con 0x7f77f40b1730 2024-03-25T18:09:24.573 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.572+0000 7f77e9ffb640 1 -- 172.21.15.156:0/4048582119 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77ec002020 con 0x7f77f40b1730 2024-03-25T18:09:24.573 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.572+0000 7f77e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:24.573 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.572+0000 7f77e9ffb640 1 -- 172.21.15.156:0/4048582119 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f77ec004fb0 con 0x7f77f40b1730 2024-03-25T18:09:24.573 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.572+0000 7f77e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:24.574 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.572+0000 7f77e9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:24.574 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.572+0000 7f77e9ffb640 10 monclient: dump: 2024-03-25T18:09:24.574 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:24.574 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:24.574 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:24.574 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:24.574 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:24.574 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:24.574 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:24.574 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:24.574 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:24.574 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:24.574 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.572+0000 7f77eb7fe640 4 set_mon_vals no callback set 2024-03-25T18:09:24.575 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.573+0000 7f77e9ffb640 1 -- 172.21.15.156:0/4048582119 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f77ec01af80 con 0x7f77f40b1730 2024-03-25T18:09:24.575 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.574+0000 7f77f8c91640 10 monclient: get_auth_request con 0x7f77d806e0f0 auth_method 0 2024-03-25T18:09:24.575 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.574+0000 7f77e9ffb640 10 client.4270.objecter ms_handle_connect 0x7f77d806e0f0 2024-03-25T18:09:24.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.578+0000 7f77e9ffb640 1 -- 172.21.15.156:0/4048582119 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f77ec054b10 con 0x7f77f40b1730 2024-03-25T18:09:24.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.578+0000 7f77e9ffb640 10 client.4270.objecter ms_dispatch 0x7f77f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:24.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.578+0000 7f77e9ffb640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:24.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.578+0000 7f77e9ffb640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:24.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.578+0000 7f77e9ffb640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-25T18:09:24.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.578+0000 7f77e9ffb640 1 -- 172.21.15.156:0/4048582119 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f77ec050450 con 0x7f77f40b1730 2024-03-25T18:09:24.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.578+0000 7f77e9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:24.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.578+0000 7f77e9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:24.818 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.818+0000 7f77fb6f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:24.819 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.818+0000 7f77fb6f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:24.819 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.818+0000 7f77fb6f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:09:24.819 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.818+0000 7f77fb6f6640 1 -- 172.21.15.156:0/4048582119 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77f40627b0 con 0x7f77f40b1730 2024-03-25T18:09:24.852 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.851+0000 7f77e9ffb640 1 -- 172.21.15.156:0/4048582119 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f77ec001590 con 0x7f77f40b1730 2024-03-25T18:09:24.852 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.851+0000 7f77e9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:24.852 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.851+0000 7f77e9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:24.852 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:09:24.852 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:24.855 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.854+0000 7f77fb6f6640 10 librados: watch_flush enter 2024-03-25T18:09:24.856 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.854+0000 7f77fb6f6640 10 librados: watch_flush exit 2024-03-25T18:09:24.856 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.854+0000 7f77fb6f6640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-25T18:09:24.856 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.854+0000 7f77fb6f6640 10 client.4270.objecter successfully canceled tick 2024-03-25T18:09:24.856 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.854+0000 7f77fb6f6640 1 -- 172.21.15.156:0/4048582119 >> v1:172.21.15.194:6800/2946427713 conn(0x7f77d806e0f0 legacy=0x7f77d80705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:24.856 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.854+0000 7f77fb6f6640 10 monclient: shutdown 2024-03-25T18:09:24.856 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.854+0000 7f77fb6f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:24.856 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.854+0000 7f77fb6f6640 1 -- 172.21.15.156:0/4048582119 >> v1:172.21.15.194:6789/0 conn(0x7f77f40b1730 legacy=0x7f77f41dac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:24.856 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.854+0000 7f77fb6f6640 1 -- 172.21.15.156:0/4048582119 shutdown_connections 2024-03-25T18:09:24.856 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.854+0000 7f77fb6f6640 1 -- 172.21.15.156:0/4048582119 >> 172.21.15.156:0/4048582119 conn(0x7f77f40617a0 msgr2=0x7f77f40b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:24.856 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.854+0000 7f77fb6f6640 1 -- 172.21.15.156:0/4048582119 shutdown_connections 2024-03-25T18:09:24.856 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.854+0000 7f77fb6f6640 1 -- 172.21.15.156:0/4048582119 wait complete. 2024-03-25T18:09:24.856 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:24.855+0000 7f77fb6f6640 1 librados: shutdown 2024-03-25T18:09:24.869 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:31.675 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:31.793 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.792+0000 7ff34bb00640 10 monclient: get_monmap_and_config 2024-03-25T18:09:31.793 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.792+0000 7ff34bb00640 10 monclient: build_initial_monmap 2024-03-25T18:09:31.793 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.792+0000 7ff34bb00640 10 monclient: monmap: 2024-03-25T18:09:31.793 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:09:31.793 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:31.793 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:09:31.793054+0000 2024-03-25T18:09:31.793 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:09:31.793054+0000 2024-03-25T18:09:31.793 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:31.793 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:31.793 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:09:31.794 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:09:31.794 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:09:31.794 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:31.794 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.792+0000 7ff34bb00640 1 Processor -- start 2024-03-25T18:09:31.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.792+0000 7ff34bb00640 1 -- start start 2024-03-25T18:09:31.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.792+0000 7ff34bb00640 10 monclient: init 2024-03-25T18:09:31.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.793+0000 7ff34bb00640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:31.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.793+0000 7ff34bb00640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:09:31.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.793+0000 7ff34bb00640 10 monclient(hunting): picked mon.noname-b con 0x7ff3440a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:09:31.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.793+0000 7ff34bb00640 10 monclient(hunting): picked mon.noname-c con 0x7ff34413f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:09:31.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.793+0000 7ff34bb00640 10 monclient(hunting): picked mon.noname-a con 0x7ff3441430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:09:31.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.793+0000 7ff34bb00640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff344142520 con 0x7ff3441430c0 2024-03-25T18:09:31.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.793+0000 7ff34bb00640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3440b28a0 con 0x7ff34413f710 2024-03-25T18:09:31.796 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.793+0000 7ff34bb00640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3440b2a30 con 0x7ff3440a98f0 2024-03-25T18:09:31.796 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.793+0000 7ff34bb00640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:31.796 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.793+0000 7ff34bb00640 10 monclient(hunting): authenticate will time out at 7053.104491s 2024-03-25T18:09:31.801 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.800+0000 7ff34909b640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7ff34413f710 0x7ff3441429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:60010/0 (socket says 172.21.15.156:60010) 2024-03-25T18:09:31.801 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.800+0000 7ff34909b640 1 -- 172.21.15.156:0/959862824 learned_addr learned my addr 172.21.15.156:0/959862824 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:09:31.821 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.820+0000 7ff34889a640 1 -- 172.21.15.156:0/959862824 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1235664599 0 0) 0x7ff3440b2a30 con 0x7ff3440a98f0 2024-03-25T18:09:31.821 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.820+0000 7ff34889a640 10 monclient(hunting): _negotiate 2024-03-25T18:09:31.821 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.820+0000 7ff34889a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:31.821 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.820+0000 7ff34889a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:31.821 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.820+0000 7ff34889a640 10 monclient(hunting): my global_id is 4283 2024-03-25T18:09:31.821 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.820+0000 7ff34889a640 1 -- 172.21.15.156:0/959862824 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff330003520 con 0x7ff3440a98f0 2024-03-25T18:09:31.826 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.826+0000 7ff34889a640 1 -- 172.21.15.156:0/959862824 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3448695833 0 0) 0x7ff344142520 con 0x7ff3441430c0 2024-03-25T18:09:31.826 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.826+0000 7ff34889a640 10 monclient(hunting): _negotiate 2024-03-25T18:09:31.827 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.826+0000 7ff34889a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:31.827 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.826+0000 7ff34889a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:31.827 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.826+0000 7ff34889a640 10 monclient(hunting): my global_id is 4275 2024-03-25T18:09:31.827 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.826+0000 7ff34889a640 1 -- 172.21.15.156:0/959862824 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3440b2a30 con 0x7ff3441430c0 2024-03-25T18:09:31.830 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.829+0000 7ff34889a640 1 -- 172.21.15.156:0/959862824 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 465967070 0 0) 0x7ff3440b28a0 con 0x7ff34413f710 2024-03-25T18:09:31.830 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.829+0000 7ff34889a640 10 monclient(hunting): _negotiate 2024-03-25T18:09:31.830 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.829+0000 7ff34889a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:31.830 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.829+0000 7ff34889a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:31.830 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.829+0000 7ff34889a640 10 monclient(hunting): my global_id is 4273 2024-03-25T18:09:31.830 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.829+0000 7ff34889a640 1 -- 172.21.15.156:0/959862824 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff344142520 con 0x7ff34413f710 2024-03-25T18:09:31.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.845+0000 7ff34889a640 1 -- 172.21.15.156:0/959862824 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3019465374 0 0) 0x7ff344142520 con 0x7ff34413f710 2024-03-25T18:09:31.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.845+0000 7ff34889a640 1 -- 172.21.15.156:0/959862824 >> v1:172.21.15.156:6790/0 conn(0x7ff3440a98f0 legacy=0x7ff3440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:31.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.845+0000 7ff34889a640 1 -- 172.21.15.156:0/959862824 >> v1:172.21.15.156:6789/0 conn(0x7ff3441430c0 legacy=0x7ff34414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:31.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.845+0000 7ff34889a640 10 monclient: _finish_hunting 0 2024-03-25T18:09:31.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.845+0000 7ff34889a640 1 monclient: found mon.noname-c 2024-03-25T18:09:31.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.845+0000 7ff34889a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:31.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.845+0000 7ff34889a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:09:31.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.845+0000 7ff34889a640 1 -- 172.21.15.156:0/959862824 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3440b44c0 con 0x7ff34413f710 2024-03-25T18:09:31.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.845+0000 7ff34889a640 10 monclient: _finish_auth 0 2024-03-25T18:09:31.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.845+0000 7ff34889a640 10 monclient: _check_auth_tickets 2024-03-25T18:09:31.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.845+0000 7ff34889a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:31.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.845+0000 7ff34889a640 1 -- 172.21.15.156:0/959862824 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7ff3340015b0 con 0x7ff34413f710 2024-03-25T18:09:31.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.845+0000 7ff34889a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:31.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.845+0000 7ff34889a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:31.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.845+0000 7ff34889a640 10 monclient: dump: 2024-03-25T18:09:31.847 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:31.847 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:31.847 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:31.847 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:31.847 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:31.847 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:31.847 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:31.847 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:31.848 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:31.848 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:31.848 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.845+0000 7ff34bb00640 5 monclient: authenticate success, global_id 4273 2024-03-25T18:09:31.848 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.845+0000 7ff34bb00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:31.855 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.854+0000 7ff34889a640 1 -- 172.21.15.156:0/959862824 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff334001710 con 0x7ff34413f710 2024-03-25T18:09:31.855 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.854+0000 7ff34889a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:31.855 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.854+0000 7ff34889a640 1 -- 172.21.15.156:0/959862824 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7ff334001b60 con 0x7ff34413f710 2024-03-25T18:09:31.855 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.854+0000 7ff34bb00640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:31.855 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.854+0000 7ff34bb00640 4 set_mon_vals no callback set 2024-03-25T18:09:31.855 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.854+0000 7ff34889a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:31.855 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.854+0000 7ff34889a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:31.855 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.854+0000 7ff34889a640 10 monclient: dump: 2024-03-25T18:09:31.855 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:31.855 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:31.855 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:31.855 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:31.855 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:31.856 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:31.856 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:31.856 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:31.856 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:31.856 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:31.857 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.855+0000 7ff34bb00640 10 monclient: shutdown 2024-03-25T18:09:31.857 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.855+0000 7ff34bb00640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:31.857 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.855+0000 7ff34bb00640 1 -- 172.21.15.156:0/959862824 >> v1:172.21.15.194:6789/0 conn(0x7ff34413f710 legacy=0x7ff3441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:31.857 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.855+0000 7ff34bb00640 1 -- 172.21.15.156:0/959862824 shutdown_connections 2024-03-25T18:09:31.857 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.855+0000 7ff34bb00640 1 -- 172.21.15.156:0/959862824 >> 172.21.15.156:0/959862824 conn(0x7ff3440b7ea0 msgr2=0x7ff3440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:31.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.855+0000 7ff34bb00640 1 -- 172.21.15.156:0/959862824 shutdown_connections 2024-03-25T18:09:31.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.855+0000 7ff34bb00640 1 -- 172.21.15.156:0/959862824 wait complete. 2024-03-25T18:09:31.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.856+0000 7ff34bb00640 10 monclient: build_initial_monmap 2024-03-25T18:09:31.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.856+0000 7ff34bb00640 10 monclient: monmap: 2024-03-25T18:09:31.858 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:09:31.858 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:31.858 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:09:31.858 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:09:31.858 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:31.858 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:09:31.858 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:09:31.858 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:09:31.858 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:09:31.858 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:31.859 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.856+0000 7ff34bb00640 1 librados: starting msgr at 2024-03-25T18:09:31.859 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.856+0000 7ff34bb00640 1 librados: starting objecter 2024-03-25T18:09:31.859 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.856+0000 7ff34bb00640 1 Processor -- start 2024-03-25T18:09:31.859 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.856+0000 7ff34bb00640 1 -- start start 2024-03-25T18:09:31.859 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.856+0000 7ff34bb00640 1 librados: setting wanted keys 2024-03-25T18:09:31.859 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.856+0000 7ff34bb00640 1 librados: calling monclient init 2024-03-25T18:09:31.859 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.856+0000 7ff34bb00640 10 monclient: init 2024-03-25T18:09:31.860 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.857+0000 7ff34bb00640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:31.860 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.857+0000 7ff34bb00640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:09:31.860 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.857+0000 7ff34bb00640 10 monclient(hunting): picked mon.noname-b con 0x7ff3440a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:09:31.860 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.857+0000 7ff34bb00640 10 monclient(hunting): picked mon.noname-a con 0x7ff34413f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:09:31.860 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.857+0000 7ff34bb00640 10 monclient(hunting): picked mon.noname-c con 0x7ff3441430c0 addr v1:172.21.15.156:6790/0 2024-03-25T18:09:31.860 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.857+0000 7ff34bb00640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3441db7f0 con 0x7ff34413f710 2024-03-25T18:09:31.860 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.857+0000 7ff34bb00640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3440abd70 con 0x7ff3440a98f0 2024-03-25T18:09:31.861 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.857+0000 7ff34bb00640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff344138ca0 con 0x7ff3441430c0 2024-03-25T18:09:31.861 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.857+0000 7ff34bb00640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:31.861 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.857+0000 7ff34bb00640 10 monclient(hunting): authenticate will time out at 7053.168514s 2024-03-25T18:09:31.874 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.874+0000 7ff34989c640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7ff3440a98f0 0x7ff344137c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:60080/0 (socket says 172.21.15.156:60080) 2024-03-25T18:09:31.875 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.874+0000 7ff34989c640 1 -- 172.21.15.156:0/2319058861 learned_addr learned my addr 172.21.15.156:0/2319058861 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:09:31.876 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.875+0000 7ff33a7fc640 10 client.?.objecter ms_handle_connect 0x7ff3440a98f0 2024-03-25T18:09:31.876 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.875+0000 7ff33a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:31.877 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.877+0000 7ff33a7fc640 10 client.?.objecter ms_handle_connect 0x7ff34413f710 2024-03-25T18:09:31.878 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.877+0000 7ff33a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:31.883 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.882+0000 7ff33a7fc640 10 client.?.objecter ms_handle_connect 0x7ff3441430c0 2024-03-25T18:09:31.883 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.882+0000 7ff33a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:31.892 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.892+0000 7ff33a7fc640 1 -- 172.21.15.156:0/2319058861 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4047537409 0 0) 0x7ff3440abd70 con 0x7ff3440a98f0 2024-03-25T18:09:31.893 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.892+0000 7ff33a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:31.893 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.892+0000 7ff33a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:31.893 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.892+0000 7ff33a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:31.893 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.892+0000 7ff33a7fc640 10 monclient(hunting): my global_id is 4276 2024-03-25T18:09:31.893 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.892+0000 7ff33a7fc640 1 -- 172.21.15.156:0/2319058861 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff328004eb0 con 0x7ff3440a98f0 2024-03-25T18:09:31.899 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.898+0000 7ff33a7fc640 1 -- 172.21.15.156:0/2319058861 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 403877233 0 0) 0x7ff344138ca0 con 0x7ff3441430c0 2024-03-25T18:09:31.899 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.898+0000 7ff33a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:31.899 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.898+0000 7ff33a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:31.899 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.898+0000 7ff33a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:31.899 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.898+0000 7ff33a7fc640 10 monclient(hunting): my global_id is 4286 2024-03-25T18:09:31.899 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.898+0000 7ff33a7fc640 1 -- 172.21.15.156:0/2319058861 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3440abd70 con 0x7ff3441430c0 2024-03-25T18:09:31.902 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.901+0000 7ff33a7fc640 1 -- 172.21.15.156:0/2319058861 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449738706 0 0) 0x7ff3441db7f0 con 0x7ff34413f710 2024-03-25T18:09:31.902 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.901+0000 7ff33a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:31.902 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.901+0000 7ff33a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:31.902 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.901+0000 7ff33a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:31.902 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.901+0000 7ff33a7fc640 10 monclient(hunting): my global_id is 4278 2024-03-25T18:09:31.902 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.901+0000 7ff33a7fc640 1 -- 172.21.15.156:0/2319058861 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff344138ca0 con 0x7ff34413f710 2024-03-25T18:09:31.911 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.910+0000 7ff33a7fc640 1 -- 172.21.15.156:0/2319058861 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2667852171 0 0) 0x7ff3440abd70 con 0x7ff3441430c0 2024-03-25T18:09:31.911 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.910+0000 7ff33a7fc640 1 -- 172.21.15.156:0/2319058861 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3441db7f0 con 0x7ff3441430c0 2024-03-25T18:09:31.911 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.910+0000 7ff33a7fc640 1 -- 172.21.15.156:0/2319058861 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7ff33c001b90 con 0x7ff3441430c0 2024-03-25T18:09:31.911 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.910+0000 7ff33a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:31.911 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.910+0000 7ff33a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:31.911 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.910+0000 7ff33a7fc640 10 monclient(hunting): dump: 2024-03-25T18:09:31.912 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:31.912 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:31.912 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:31.912 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:31.912 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:31.912 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:31.912 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:31.912 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:31.912 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:31.912 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:31.912 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.910+0000 7ff33a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:31.913 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.911+0000 7ff33a7fc640 1 -- 172.21.15.156:0/2319058861 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4164683258 0 0) 0x7ff344138ca0 con 0x7ff34413f710 2024-03-25T18:09:31.913 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.911+0000 7ff33a7fc640 1 -- 172.21.15.156:0/2319058861 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3440abd70 con 0x7ff34413f710 2024-03-25T18:09:31.913 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.911+0000 7ff33a7fc640 1 -- 172.21.15.156:0/2319058861 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7ff334001a20 con 0x7ff34413f710 2024-03-25T18:09:31.914 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.911+0000 7ff33a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:31.914 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.911+0000 7ff33a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:31.914 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.911+0000 7ff33a7fc640 10 monclient(hunting): dump: 2024-03-25T18:09:31.914 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:31.914 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:31.914 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:31.914 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:31.914 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:31.914 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:31.914 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:31.914 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:31.914 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:31.914 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:31.914 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.912+0000 7ff33a7fc640 1 -- 172.21.15.156:0/2319058861 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2658942563 0 0) 0x7ff328004eb0 con 0x7ff3440a98f0 2024-03-25T18:09:31.915 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.912+0000 7ff33a7fc640 1 -- 172.21.15.156:0/2319058861 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff344138ca0 con 0x7ff3440a98f0 2024-03-25T18:09:31.915 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.912+0000 7ff33a7fc640 1 -- 172.21.15.156:0/2319058861 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7ff340001990 con 0x7ff3440a98f0 2024-03-25T18:09:31.915 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.912+0000 7ff33a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:31.915 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.912+0000 7ff33a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:31.915 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.912+0000 7ff33a7fc640 10 monclient(hunting): dump: 2024-03-25T18:09:31.915 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:31.915 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:31.915 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:31.915 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:31.915 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:31.915 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:31.915 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:31.915 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:31.915 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:31.915 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:31.924 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.923+0000 7ff33a7fc640 1 -- 172.21.15.156:0/2319058861 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2147846030 0 0) 0x7ff3440abd70 con 0x7ff34413f710 2024-03-25T18:09:31.924 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.923+0000 7ff33a7fc640 1 -- 172.21.15.156:0/2319058861 >> v1:172.21.15.156:6790/0 conn(0x7ff3441430c0 legacy=0x7ff344138590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:31.924 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.923+0000 7ff33a7fc640 1 -- 172.21.15.156:0/2319058861 >> v1:172.21.15.194:6789/0 conn(0x7ff3440a98f0 legacy=0x7ff344137c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:31.924 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.923+0000 7ff33a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:09:31.925 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.923+0000 7ff33a7fc640 1 monclient: found mon.a 2024-03-25T18:09:31.925 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.923+0000 7ff33a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:31.925 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.923+0000 7ff33a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:09:31.925 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.923+0000 7ff33a7fc640 1 -- 172.21.15.156:0/2319058861 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff344138e30 con 0x7ff34413f710 2024-03-25T18:09:31.925 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.923+0000 7ff33a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:09:31.925 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.923+0000 7ff33a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:09:31.925 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.923+0000 7ff33a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:31.925 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.923+0000 7ff34bb00640 5 monclient: authenticate success, global_id 4278 2024-03-25T18:09:31.925 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.923+0000 7ff34bb00640 10 monclient: _renew_subs 2024-03-25T18:09:31.925 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.923+0000 7ff34bb00640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:09:31.925 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.923+0000 7ff34bb00640 1 -- 172.21.15.156:0/2319058861 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3441e2c30 con 0x7ff34413f710 2024-03-25T18:09:31.925 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.923+0000 7ff34bb00640 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:31.925 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.923+0000 7ff34bb00640 10 monclient: _renew_subs 2024-03-25T18:09:31.925 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.923+0000 7ff34bb00640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:09:31.925 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.923+0000 7ff34bb00640 1 -- 172.21.15.156:0/2319058861 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3441e2fa0 con 0x7ff34413f710 2024-03-25T18:09:31.925 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.923+0000 7ff34bb00640 1 librados: init done 2024-03-25T18:09:31.926 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.924+0000 7ff34bb00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:31.926 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.924+0000 7ff34bb00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:31.926 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.924+0000 7ff34bb00640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:09:31.926 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:31.924+0000 7ff34bb00640 1 -- 172.21.15.156:0/2319058861 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3440b3ef0 con 0x7ff34413f710 2024-03-25T18:09:32.228 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.227+0000 7ff33a7fc640 1 -- 172.21.15.156:0/2319058861 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff334001ba0 con 0x7ff34413f710 2024-03-25T18:09:32.229 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.227+0000 7ff33a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:32.229 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.227+0000 7ff33a7fc640 1 -- 172.21.15.156:0/2319058861 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7ff334004c30 con 0x7ff34413f710 2024-03-25T18:09:32.229 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.227+0000 7ff33a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:32.229 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.227+0000 7ff33a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:32.229 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.227+0000 7ff33a7fc640 10 monclient: dump: 2024-03-25T18:09:32.229 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:32.229 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:32.229 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:32.229 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:32.229 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:32.229 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:32.229 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:32.229 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:32.230 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:32.230 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:32.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.227+0000 7ff33bfff640 4 set_mon_vals no callback set 2024-03-25T18:09:32.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.229+0000 7ff33a7fc640 1 -- 172.21.15.156:0/2319058861 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7ff33401ac00 con 0x7ff34413f710 2024-03-25T18:09:32.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.229+0000 7ff34989c640 10 monclient: get_auth_request con 0x7ff32806e110 auth_method 0 2024-03-25T18:09:32.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.229+0000 7ff33a7fc640 10 client.4278.objecter ms_handle_connect 0x7ff32806e110 2024-03-25T18:09:32.249 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.248+0000 7ff33a7fc640 1 -- 172.21.15.156:0/2319058861 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7ff334054770 con 0x7ff34413f710 2024-03-25T18:09:32.249 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.248+0000 7ff33a7fc640 10 client.4278.objecter ms_dispatch 0x7ff344000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:32.249 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.248+0000 7ff33a7fc640 3 client.4278.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:32.249 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.248+0000 7ff33a7fc640 3 client.4278.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:32.249 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.248+0000 7ff33a7fc640 20 client.4278.objecter dump_active .. 0 homeless 2024-03-25T18:09:32.249 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.248+0000 7ff33a7fc640 1 -- 172.21.15.156:0/2319058861 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff3340500b0 con 0x7ff34413f710 2024-03-25T18:09:32.250 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.248+0000 7ff33a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:32.250 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.248+0000 7ff33a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:32.468 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.467+0000 7ff34bb00640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:32.468 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.467+0000 7ff34bb00640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:32.468 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.467+0000 7ff34bb00640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:09:32.468 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.467+0000 7ff34bb00640 1 -- 172.21.15.156:0/2319058861 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3441dadd0 con 0x7ff34413f710 2024-03-25T18:09:32.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.507+0000 7ff33a7fc640 1 -- 172.21.15.156:0/2319058861 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7ff3340521b0 con 0x7ff34413f710 2024-03-25T18:09:32.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.507+0000 7ff33a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:32.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.507+0000 7ff33a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:32.508 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:09:32.509 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:32.511 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.509+0000 7ff34bb00640 10 librados: watch_flush enter 2024-03-25T18:09:32.511 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.510+0000 7ff34bb00640 10 librados: watch_flush exit 2024-03-25T18:09:32.511 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.510+0000 7ff34bb00640 20 client.4278.objecter shutdown clearing up homeless session... 2024-03-25T18:09:32.511 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.510+0000 7ff34bb00640 10 client.4278.objecter successfully canceled tick 2024-03-25T18:09:32.511 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.510+0000 7ff34bb00640 1 -- 172.21.15.156:0/2319058861 >> v1:172.21.15.194:6800/2946427713 conn(0x7ff32806e110 legacy=0x7ff3280705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:32.512 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.510+0000 7ff34bb00640 10 monclient: shutdown 2024-03-25T18:09:32.512 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.510+0000 7ff34bb00640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:32.512 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.510+0000 7ff34bb00640 1 -- 172.21.15.156:0/2319058861 >> v1:172.21.15.156:6789/0 conn(0x7ff34413f710 legacy=0x7ff3441db0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:32.512 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.510+0000 7ff34bb00640 1 -- 172.21.15.156:0/2319058861 shutdown_connections 2024-03-25T18:09:32.512 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.510+0000 7ff34bb00640 1 -- 172.21.15.156:0/2319058861 >> 172.21.15.156:0/2319058861 conn(0x7ff3440b7ea0 msgr2=0x7ff344074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:32.512 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.510+0000 7ff34bb00640 1 -- 172.21.15.156:0/2319058861 shutdown_connections 2024-03-25T18:09:32.512 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.510+0000 7ff34bb00640 1 -- 172.21.15.156:0/2319058861 wait complete. 2024-03-25T18:09:32.512 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:32.510+0000 7ff34bb00640 1 librados: shutdown 2024-03-25T18:09:32.522 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:33.079 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:01:39.069 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:33.083 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:01:39.074 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:33.133 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:01:39.100 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:33.137 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:01:39.104 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:33.139 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:01:39.130 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:33.149 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:01:39.139 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:33.153 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:01:39.144 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:33.195 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:01:39.163 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:33.212 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:01:39.203 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:39.327 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:39.439 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.437+0000 7f54b104e640 10 monclient: get_monmap_and_config 2024-03-25T18:09:39.439 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.437+0000 7f54b104e640 10 monclient: build_initial_monmap 2024-03-25T18:09:39.439 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.437+0000 7f54b104e640 10 monclient: monmap: 2024-03-25T18:09:39.439 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:09:39.439 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:39.439 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:09:39.439946+0000 2024-03-25T18:09:39.440 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:09:39.439946+0000 2024-03-25T18:09:39.440 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:39.440 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:39.440 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:09:39.440 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:09:39.440 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:09:39.440 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:39.441 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.438+0000 7f54b104e640 1 Processor -- start 2024-03-25T18:09:39.441 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.438+0000 7f54b104e640 1 -- start start 2024-03-25T18:09:39.441 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.438+0000 7f54b104e640 10 monclient: init 2024-03-25T18:09:39.441 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.439+0000 7f54b104e640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:39.441 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.439+0000 7f54b104e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:09:39.441 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.439+0000 7f54b104e640 10 monclient(hunting): picked mon.noname-b con 0x7f54ac0ae6e0 addr v1:172.21.15.156:6790/0 2024-03-25T18:09:39.441 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.439+0000 7f54b104e640 10 monclient(hunting): picked mon.noname-c con 0x7f54ac06f080 addr v1:172.21.15.194:6789/0 2024-03-25T18:09:39.442 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.439+0000 7f54b104e640 10 monclient(hunting): picked mon.noname-a con 0x7f54ac0704e0 addr v1:172.21.15.156:6789/0 2024-03-25T18:09:39.442 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.439+0000 7f54b104e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54ac0718d0 con 0x7f54ac0704e0 2024-03-25T18:09:39.442 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.439+0000 7f54b104e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54ac06fa30 con 0x7f54ac06f080 2024-03-25T18:09:39.442 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.439+0000 7f54b104e640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54ac070090 con 0x7f54ac0ae6e0 2024-03-25T18:09:39.442 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.439+0000 7f54b104e640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:39.442 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.439+0000 7f54b104e640 10 monclient(hunting): authenticate will time out at 7060.751274s 2024-03-25T18:09:39.453 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.451+0000 7f54aa59c640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f54ac06f080 0x7f54ac071c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:36384/0 (socket says 172.21.15.156:36384) 2024-03-25T18:09:39.453 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.451+0000 7f54aa59c640 1 -- 172.21.15.156:0/2614626256 learned_addr learned my addr 172.21.15.156:0/2614626256 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:09:39.464 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.463+0000 7f54a9d9b640 1 -- 172.21.15.156:0/2614626256 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724244564 0 0) 0x7f54ac0718d0 con 0x7f54ac0704e0 2024-03-25T18:09:39.465 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.463+0000 7f54a9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:09:39.465 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.463+0000 7f54a9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:39.465 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.463+0000 7f54a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:39.465 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.463+0000 7f54a9d9b640 10 monclient(hunting): my global_id is 4281 2024-03-25T18:09:39.465 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.463+0000 7f54a9d9b640 1 -- 172.21.15.156:0/2614626256 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5494003520 con 0x7f54ac0704e0 2024-03-25T18:09:39.465 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.464+0000 7f54a9d9b640 1 -- 172.21.15.156:0/2614626256 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459405439 0 0) 0x7f54ac06fa30 con 0x7f54ac06f080 2024-03-25T18:09:39.465 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.464+0000 7f54a9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:09:39.466 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.464+0000 7f54a9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:39.466 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.464+0000 7f54a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:39.466 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.464+0000 7f54a9d9b640 10 monclient(hunting): my global_id is 4279 2024-03-25T18:09:39.466 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.464+0000 7f54a9d9b640 1 -- 172.21.15.156:0/2614626256 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54ac0718d0 con 0x7f54ac06f080 2024-03-25T18:09:39.468 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.466+0000 7f54a9d9b640 1 -- 172.21.15.156:0/2614626256 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397388032 0 0) 0x7f54ac070090 con 0x7f54ac0ae6e0 2024-03-25T18:09:39.468 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.466+0000 7f54a9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:09:39.468 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.466+0000 7f54a9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:39.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.466+0000 7f54a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:39.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.466+0000 7f54a9d9b640 10 monclient(hunting): my global_id is 4289 2024-03-25T18:09:39.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.466+0000 7f54a9d9b640 1 -- 172.21.15.156:0/2614626256 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54ac06fa30 con 0x7f54ac0ae6e0 2024-03-25T18:09:39.477 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.475+0000 7f54a9d9b640 1 -- 172.21.15.156:0/2614626256 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1515394318 0 0) 0x7f5494003520 con 0x7f54ac0704e0 2024-03-25T18:09:39.477 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.475+0000 7f54a9d9b640 1 -- 172.21.15.156:0/2614626256 >> v1:172.21.15.156:6790/0 conn(0x7f54ac0ae6e0 legacy=0x7f54ac13c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:39.477 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.475+0000 7f54a9d9b640 1 -- 172.21.15.156:0/2614626256 >> v1:172.21.15.194:6789/0 conn(0x7f54ac06f080 legacy=0x7f54ac071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:39.477 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.475+0000 7f54a9d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:09:39.477 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.475+0000 7f54a9d9b640 1 monclient: found mon.noname-a 2024-03-25T18:09:39.477 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.475+0000 7f54a9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:39.477 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.475+0000 7f54a9d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:09:39.477 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.475+0000 7f54a9d9b640 1 -- 172.21.15.156:0/2614626256 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54ac06f510 con 0x7f54ac0704e0 2024-03-25T18:09:39.477 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.475+0000 7f54a9d9b640 10 monclient: _finish_auth 0 2024-03-25T18:09:39.477 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.475+0000 7f54a9d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:09:39.477 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.475+0000 7f54a9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:39.477 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.475+0000 7f54a9d9b640 1 -- 172.21.15.156:0/2614626256 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f54a4001a10 con 0x7f54ac0704e0 2024-03-25T18:09:39.477 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.475+0000 7f54b104e640 5 monclient: authenticate success, global_id 4281 2024-03-25T18:09:39.477 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.475+0000 7f54b104e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:39.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.475+0000 7f54a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:39.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.475+0000 7f54a9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:39.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.475+0000 7f54a9d9b640 10 monclient: dump: 2024-03-25T18:09:39.478 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:39.478 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:39.478 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:39.478 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:39.478 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:39.478 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:39.478 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:39.478 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:39.478 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:39.478 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:39.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.475+0000 7f54b104e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:39.479 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.478+0000 7f54a9d9b640 1 -- 172.21.15.156:0/2614626256 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54a4001b70 con 0x7f54ac0704e0 2024-03-25T18:09:39.479 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.478+0000 7f54a9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:39.479 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.478+0000 7f54a9d9b640 1 -- 172.21.15.156:0/2614626256 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f54a4001f80 con 0x7f54ac0704e0 2024-03-25T18:09:39.479 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.478+0000 7f54b104e640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:39.480 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.478+0000 7f54b104e640 4 set_mon_vals no callback set 2024-03-25T18:09:39.480 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.478+0000 7f54a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:39.480 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.478+0000 7f54a9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:39.480 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.478+0000 7f54a9d9b640 10 monclient: dump: 2024-03-25T18:09:39.480 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:39.480 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:39.480 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:39.480 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:39.480 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:39.480 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:39.480 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:39.480 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:39.480 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:39.480 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:39.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.479+0000 7f54b104e640 10 monclient: shutdown 2024-03-25T18:09:39.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.479+0000 7f54b104e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:39.482 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.479+0000 7f54b104e640 1 -- 172.21.15.156:0/2614626256 >> v1:172.21.15.156:6789/0 conn(0x7f54ac0704e0 legacy=0x7f54ac14a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:39.482 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.479+0000 7f54b104e640 1 -- 172.21.15.156:0/2614626256 shutdown_connections 2024-03-25T18:09:39.482 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.479+0000 7f54b104e640 1 -- 172.21.15.156:0/2614626256 >> 172.21.15.156:0/2614626256 conn(0x7f54ac06b630 msgr2=0x7f54ac0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:39.482 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.479+0000 7f54b104e640 1 -- 172.21.15.156:0/2614626256 shutdown_connections 2024-03-25T18:09:39.482 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.479+0000 7f54b104e640 1 -- 172.21.15.156:0/2614626256 wait complete. 2024-03-25T18:09:39.482 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.480+0000 7f54b104e640 10 monclient: build_initial_monmap 2024-03-25T18:09:39.482 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.480+0000 7f54b104e640 10 monclient: monmap: 2024-03-25T18:09:39.482 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:09:39.482 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:39.482 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:09:39.482 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:09:39.482 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:39.482 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:09:39.482 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:09:39.482 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:09:39.483 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:09:39.483 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:39.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.480+0000 7f54b104e640 1 librados: starting msgr at 2024-03-25T18:09:39.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.480+0000 7f54b104e640 1 librados: starting objecter 2024-03-25T18:09:39.484 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.480+0000 7f54b104e640 1 Processor -- start 2024-03-25T18:09:39.484 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.480+0000 7f54b104e640 1 -- start start 2024-03-25T18:09:39.484 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.480+0000 7f54b104e640 1 librados: setting wanted keys 2024-03-25T18:09:39.484 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.480+0000 7f54b104e640 1 librados: calling monclient init 2024-03-25T18:09:39.484 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.480+0000 7f54b104e640 10 monclient: init 2024-03-25T18:09:39.484 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.481+0000 7f54b104e640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:39.484 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.481+0000 7f54b104e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:09:39.484 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.481+0000 7f54b104e640 10 monclient(hunting): picked mon.noname-c con 0x7f54ac0704e0 addr v1:172.21.15.156:6790/0 2024-03-25T18:09:39.484 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.481+0000 7f54b104e640 10 monclient(hunting): picked mon.noname-a con 0x7f54ac0ae6e0 addr v1:172.21.15.156:6789/0 2024-03-25T18:09:39.484 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.481+0000 7f54b104e640 10 monclient(hunting): picked mon.noname-b con 0x7f54ac13eb00 addr v1:172.21.15.194:6789/0 2024-03-25T18:09:39.484 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.481+0000 7f54b104e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54ac1dbfd0 con 0x7f54ac0ae6e0 2024-03-25T18:09:39.484 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.481+0000 7f54b104e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54ac0a97f0 con 0x7f54ac13eb00 2024-03-25T18:09:39.484 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.481+0000 7f54b104e640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54ac0a9ad0 con 0x7f54ac0704e0 2024-03-25T18:09:39.485 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.481+0000 7f54b104e640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:39.485 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.481+0000 7f54b104e640 10 monclient(hunting): authenticate will time out at 7060.793349s 2024-03-25T18:09:39.493 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.491+0000 7f54aad9d640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f54ac0704e0 0x7f54ac0a9ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:58804/0 (socket says 172.21.15.156:58804) 2024-03-25T18:09:39.493 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.491+0000 7f54aad9d640 1 -- 172.21.15.156:0/2578080634 learned_addr learned my addr 172.21.15.156:0/2578080634 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:09:39.496 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.495+0000 7f54937fe640 10 client.?.objecter ms_handle_connect 0x7f54ac0704e0 2024-03-25T18:09:39.496 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.495+0000 7f54937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:39.500 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.499+0000 7f54937fe640 10 client.?.objecter ms_handle_connect 0x7f54ac0ae6e0 2024-03-25T18:09:39.500 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.499+0000 7f54937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:39.504 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.502+0000 7f54937fe640 10 client.?.objecter ms_handle_connect 0x7f54ac13eb00 2024-03-25T18:09:39.504 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.502+0000 7f54937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:39.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.506+0000 7f54937fe640 1 -- 172.21.15.156:0/2578080634 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 522320566 0 0) 0x7f54ac0a9ad0 con 0x7f54ac0704e0 2024-03-25T18:09:39.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.506+0000 7f54937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:39.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.506+0000 7f54937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:39.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.506+0000 7f54937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:39.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.506+0000 7f54937fe640 10 monclient(hunting): my global_id is 4292 2024-03-25T18:09:39.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.506+0000 7f54937fe640 1 -- 172.21.15.156:0/2578080634 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5498004eb0 con 0x7f54ac0704e0 2024-03-25T18:09:39.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.507+0000 7f54937fe640 1 -- 172.21.15.156:0/2578080634 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802596048 0 0) 0x7f54ac0a97f0 con 0x7f54ac13eb00 2024-03-25T18:09:39.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.507+0000 7f54937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:39.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.507+0000 7f54937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:39.510 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.507+0000 7f54937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:39.510 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.507+0000 7f54937fe640 10 monclient(hunting): my global_id is 4282 2024-03-25T18:09:39.510 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.507+0000 7f54937fe640 1 -- 172.21.15.156:0/2578080634 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54ac0a9ad0 con 0x7f54ac13eb00 2024-03-25T18:09:39.520 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.518+0000 7f54937fe640 1 -- 172.21.15.156:0/2578080634 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3743332312 0 0) 0x7f54ac0a9ad0 con 0x7f54ac13eb00 2024-03-25T18:09:39.520 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.518+0000 7f54937fe640 1 -- 172.21.15.156:0/2578080634 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54ac0a97f0 con 0x7f54ac13eb00 2024-03-25T18:09:39.520 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.518+0000 7f54937fe640 1 -- 172.21.15.156:0/2578080634 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f54a4001e80 con 0x7f54ac13eb00 2024-03-25T18:09:39.520 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.518+0000 7f54937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:39.520 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.518+0000 7f54937fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:39.520 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.518+0000 7f54937fe640 10 monclient(hunting): dump: 2024-03-25T18:09:39.520 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:39.520 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:39.520 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:39.521 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:39.521 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:39.521 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:39.521 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:39.521 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:39.521 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:39.521 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:39.521 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.518+0000 7f54937fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:39.521 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.518+0000 7f54937fe640 1 -- 172.21.15.156:0/2578080634 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459167802 0 0) 0x7f54ac1dbfd0 con 0x7f54ac0ae6e0 2024-03-25T18:09:39.521 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.518+0000 7f54937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:39.521 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.518+0000 7f54937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:39.521 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.518+0000 7f54937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:39.521 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.518+0000 7f54937fe640 10 monclient(hunting): my global_id is 4284 2024-03-25T18:09:39.521 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.518+0000 7f54937fe640 1 -- 172.21.15.156:0/2578080634 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54ac0a9ad0 con 0x7f54ac0ae6e0 2024-03-25T18:09:39.522 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.519+0000 7f54937fe640 1 -- 172.21.15.156:0/2578080634 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2206769925 0 0) 0x7f5498004eb0 con 0x7f54ac0704e0 2024-03-25T18:09:39.522 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.519+0000 7f54937fe640 1 -- 172.21.15.156:0/2578080634 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54ac1dbfd0 con 0x7f54ac0704e0 2024-03-25T18:09:39.522 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.519+0000 7f54937fe640 1 -- 172.21.15.156:0/2578080634 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f549c001970 con 0x7f54ac0704e0 2024-03-25T18:09:39.522 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.519+0000 7f54937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:39.522 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.519+0000 7f54937fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:39.522 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.519+0000 7f54937fe640 10 monclient(hunting): dump: 2024-03-25T18:09:39.522 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:39.522 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:39.522 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:39.522 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:39.522 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:39.523 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:39.523 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:39.523 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:39.523 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:39.523 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:39.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.524+0000 7f54937fe640 1 -- 172.21.15.156:0/2578080634 <== mon.2 v1:172.21.15.156:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 319331419 0 0) 0x7f54ac1dbfd0 con 0x7f54ac0704e0 2024-03-25T18:09:39.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.524+0000 7f54937fe640 1 -- 172.21.15.156:0/2578080634 >> v1:172.21.15.194:6789/0 conn(0x7f54ac13eb00 legacy=0x7f54ac1db8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:39.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.524+0000 7f54937fe640 1 -- 172.21.15.156:0/2578080634 >> v1:172.21.15.156:6789/0 conn(0x7f54ac0ae6e0 legacy=0x7f54ac1db1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:39.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.524+0000 7f54937fe640 10 monclient: _finish_hunting 0 2024-03-25T18:09:39.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.524+0000 7f54937fe640 1 monclient: found mon.c 2024-03-25T18:09:39.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.524+0000 7f54937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:39.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.524+0000 7f54937fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:09:39.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.524+0000 7f54937fe640 1 -- 172.21.15.156:0/2578080634 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54ac141f70 con 0x7f54ac0704e0 2024-03-25T18:09:39.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.524+0000 7f54937fe640 10 monclient: _finish_auth 0 2024-03-25T18:09:39.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.524+0000 7f54937fe640 10 monclient: _check_auth_tickets 2024-03-25T18:09:39.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.524+0000 7f54937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:39.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.524+0000 7f54b104e640 5 monclient: authenticate success, global_id 4292 2024-03-25T18:09:39.527 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.524+0000 7f54b104e640 10 monclient: _renew_subs 2024-03-25T18:09:39.527 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.524+0000 7f54b104e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:09:39.527 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.524+0000 7f54b104e640 1 -- 172.21.15.156:0/2578080634 --> v1:172.21.15.156:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54ac0aa600 con 0x7f54ac0704e0 2024-03-25T18:09:39.527 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.524+0000 7f54b104e640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:39.527 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.524+0000 7f54b104e640 10 monclient: _renew_subs 2024-03-25T18:09:39.527 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.524+0000 7f54b104e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:09:39.527 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.524+0000 7f54b104e640 1 -- 172.21.15.156:0/2578080634 --> v1:172.21.15.156:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54ac1415c0 con 0x7f54ac0704e0 2024-03-25T18:09:39.527 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.524+0000 7f54b104e640 1 librados: init done 2024-03-25T18:09:39.527 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.525+0000 7f54b104e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:39.528 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.525+0000 7f54b104e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:39.528 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.525+0000 7f54b104e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:09:39.528 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.525+0000 7f54b104e640 1 -- 172.21.15.156:0/2578080634 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54ac0addd0 con 0x7f54ac0704e0 2024-03-25T18:09:39.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.546+0000 7f54937fe640 1 -- 172.21.15.156:0/2578080634 <== mon.2 v1:172.21.15.156:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f549c003610 con 0x7f54ac0704e0 2024-03-25T18:09:39.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.546+0000 7f54937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:39.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.546+0000 7f54937fe640 1 -- 172.21.15.156:0/2578080634 <== mon.2 v1:172.21.15.156:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f549c0039a0 con 0x7f54ac0704e0 2024-03-25T18:09:39.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.546+0000 7f54937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:39.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.546+0000 7f54937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:39.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.546+0000 7f54937fe640 10 monclient: dump: 2024-03-25T18:09:39.548 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:39.548 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:39.548 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:39.548 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:39.548 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:39.548 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:39.548 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:39.548 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:39.548 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:39.549 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:39.549 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.546+0000 7f54a959a640 4 set_mon_vals no callback set 2024-03-25T18:09:39.549 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.547+0000 7f54937fe640 1 -- 172.21.15.156:0/2578080634 <== mon.2 v1:172.21.15.156:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f549c003bc0 con 0x7f54ac0704e0 2024-03-25T18:09:39.549 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.547+0000 7f54aa59c640 10 monclient: get_auth_request con 0x7f549806df50 auth_method 0 2024-03-25T18:09:39.549 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.548+0000 7f54937fe640 10 client.4292.objecter ms_handle_connect 0x7f549806df50 2024-03-25T18:09:39.552 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.550+0000 7f54937fe640 1 -- 172.21.15.156:0/2578080634 <== mon.2 v1:172.21.15.156:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f549c054810 con 0x7f54ac0704e0 2024-03-25T18:09:39.553 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.550+0000 7f54937fe640 10 client.4292.objecter ms_dispatch 0x7f54ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:39.553 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.550+0000 7f54937fe640 3 client.4292.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:39.553 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.551+0000 7f54937fe640 3 client.4292.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:39.553 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.551+0000 7f54937fe640 20 client.4292.objecter dump_active .. 0 homeless 2024-03-25T18:09:39.827 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.827+0000 7f54937fe640 1 -- 172.21.15.156:0/2578080634 <== mon.2 v1:172.21.15.156:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f549c050320 con 0x7f54ac0704e0 2024-03-25T18:09:39.828 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.827+0000 7f54937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:39.828 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:39.827+0000 7f54937fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:40.081 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:40.080+0000 7f54b104e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:40.081 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:40.080+0000 7f54b104e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:40.081 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:40.080+0000 7f54b104e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:09:40.081 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:40.080+0000 7f54b104e640 1 -- 172.21.15.156:0/2578080634 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54ac074d80 con 0x7f54ac0704e0 2024-03-25T18:09:40.105 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:40.105+0000 7f54937fe640 1 -- 172.21.15.156:0/2578080634 <== mon.2 v1:172.21.15.156:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f54ac074d80 con 0x7f54ac0704e0 2024-03-25T18:09:40.105 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:40.105+0000 7f54937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:40.105 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:40.105+0000 7f54937fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:40.106 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:09:40.106 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:40.108 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:40.107+0000 7f54b104e640 10 librados: watch_flush enter 2024-03-25T18:09:40.109 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:40.107+0000 7f54b104e640 10 librados: watch_flush exit 2024-03-25T18:09:40.109 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:40.107+0000 7f54b104e640 20 client.4292.objecter shutdown clearing up homeless session... 2024-03-25T18:09:40.109 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:40.107+0000 7f54b104e640 10 client.4292.objecter successfully canceled tick 2024-03-25T18:09:40.109 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:40.107+0000 7f54b104e640 1 -- 172.21.15.156:0/2578080634 >> v1:172.21.15.194:6800/2946427713 conn(0x7f549806df50 legacy=0x7f5498070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:40.109 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:40.107+0000 7f54b104e640 10 monclient: shutdown 2024-03-25T18:09:40.109 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:40.107+0000 7f54b104e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:40.109 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:40.107+0000 7f54b104e640 1 -- 172.21.15.156:0/2578080634 >> v1:172.21.15.156:6790/0 conn(0x7f54ac0704e0 legacy=0x7f54ac0a9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:40.109 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:40.107+0000 7f54b104e640 1 -- 172.21.15.156:0/2578080634 shutdown_connections 2024-03-25T18:09:40.109 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:40.107+0000 7f54b104e640 1 -- 172.21.15.156:0/2578080634 >> 172.21.15.156:0/2578080634 conn(0x7f54ac06b630 msgr2=0x7f54ac13c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:40.109 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:40.107+0000 7f54b104e640 1 -- 172.21.15.156:0/2578080634 shutdown_connections 2024-03-25T18:09:40.109 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:40.108+0000 7f54b104e640 1 -- 172.21.15.156:0/2578080634 wait complete. 2024-03-25T18:09:40.109 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:40.108+0000 7f54b104e640 1 librados: shutdown 2024-03-25T18:09:40.120 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:46.927 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:47.046 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.045+0000 7fc77e3cb640 10 monclient: get_monmap_and_config 2024-03-25T18:09:47.046 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.045+0000 7fc77e3cb640 10 monclient: build_initial_monmap 2024-03-25T18:09:47.046 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.045+0000 7fc77e3cb640 10 monclient: monmap: 2024-03-25T18:09:47.046 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:09:47.047 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:47.047 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:09:47.046745+0000 2024-03-25T18:09:47.047 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:09:47.046745+0000 2024-03-25T18:09:47.047 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:47.047 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:47.047 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:09:47.047 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:09:47.047 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:09:47.047 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:47.048 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.046+0000 7fc77e3cb640 1 Processor -- start 2024-03-25T18:09:47.048 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.046+0000 7fc77e3cb640 1 -- start start 2024-03-25T18:09:47.048 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.046+0000 7fc77e3cb640 10 monclient: init 2024-03-25T18:09:47.048 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.047+0000 7fc77e3cb640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:47.048 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.047+0000 7fc77e3cb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:47.048 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.047+0000 7fc77e3cb640 10 monclient(hunting): picked mon.noname-c con 0x7fc778061890 addr v1:172.21.15.194:6789/0 2024-03-25T18:09:47.048 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.047+0000 7fc77e3cb640 10 monclient(hunting): picked mon.noname-b con 0x7fc77806e740 addr v1:172.21.15.156:6790/0 2024-03-25T18:09:47.048 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.047+0000 7fc77e3cb640 10 monclient(hunting): picked mon.noname-a con 0x7fc7780ae280 addr v1:172.21.15.156:6789/0 2024-03-25T18:09:47.049 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.047+0000 7fc77e3cb640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7780ae830 con 0x7fc7780ae280 2024-03-25T18:09:47.049 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.047+0000 7fc77e3cb640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc77806f720 con 0x7fc778061890 2024-03-25T18:09:47.049 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.047+0000 7fc77e3cb640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7780ada80 con 0x7fc77806e740 2024-03-25T18:09:47.049 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.047+0000 7fc77e3cb640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:47.049 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.047+0000 7fc77e3cb640 10 monclient(hunting): authenticate will time out at 7068.358356s 2024-03-25T18:09:47.053 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.052+0000 7fc777fff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fc778061890 0x7fc77813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:40816/0 (socket says 172.21.15.156:40816) 2024-03-25T18:09:47.053 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.053+0000 7fc777fff640 1 -- 172.21.15.156:0/3403554152 learned_addr learned my addr 172.21.15.156:0/3403554152 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:09:47.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.070+0000 7fc776ffd640 1 -- 172.21.15.156:0/3403554152 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 461871690 0 0) 0x7fc7780ae830 con 0x7fc7780ae280 2024-03-25T18:09:47.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.070+0000 7fc776ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:47.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.070+0000 7fc776ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:47.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.070+0000 7fc776ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:47.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.070+0000 7fc776ffd640 10 monclient(hunting): my global_id is 4287 2024-03-25T18:09:47.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.070+0000 7fc776ffd640 1 -- 172.21.15.156:0/3403554152 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc760003520 con 0x7fc7780ae280 2024-03-25T18:09:47.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.070+0000 7fc776ffd640 1 -- 172.21.15.156:0/3403554152 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4059870557 0 0) 0x7fc77806f720 con 0x7fc778061890 2024-03-25T18:09:47.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.070+0000 7fc776ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:47.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.070+0000 7fc776ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:47.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.070+0000 7fc776ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:47.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.070+0000 7fc776ffd640 10 monclient(hunting): my global_id is 4285 2024-03-25T18:09:47.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.070+0000 7fc776ffd640 1 -- 172.21.15.156:0/3403554152 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7780ae830 con 0x7fc778061890 2024-03-25T18:09:47.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.084+0000 7fc776ffd640 1 -- 172.21.15.156:0/3403554152 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4010129123 0 0) 0x7fc7780ada80 con 0x7fc77806e740 2024-03-25T18:09:47.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.084+0000 7fc776ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:47.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.084+0000 7fc776ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:47.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.084+0000 7fc776ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:47.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.084+0000 7fc776ffd640 10 monclient(hunting): my global_id is 4295 2024-03-25T18:09:47.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.084+0000 7fc776ffd640 1 -- 172.21.15.156:0/3403554152 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc77806f720 con 0x7fc77806e740 2024-03-25T18:09:47.089 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.087+0000 7fc776ffd640 1 -- 172.21.15.156:0/3403554152 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1633518219 0 0) 0x7fc760003520 con 0x7fc7780ae280 2024-03-25T18:09:47.089 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.088+0000 7fc776ffd640 1 -- 172.21.15.156:0/3403554152 >> v1:172.21.15.156:6790/0 conn(0x7fc77806e740 legacy=0x7fc778145820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:47.089 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.088+0000 7fc776ffd640 1 -- 172.21.15.156:0/3403554152 >> v1:172.21.15.194:6789/0 conn(0x7fc778061890 legacy=0x7fc77813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:47.089 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.088+0000 7fc776ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:09:47.089 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.088+0000 7fc776ffd640 1 monclient: found mon.noname-a 2024-03-25T18:09:47.089 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.088+0000 7fc776ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:47.089 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.088+0000 7fc776ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:09:47.089 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.088+0000 7fc776ffd640 1 -- 172.21.15.156:0/3403554152 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc77813fbd0 con 0x7fc7780ae280 2024-03-25T18:09:47.089 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.088+0000 7fc776ffd640 10 monclient: _finish_auth 0 2024-03-25T18:09:47.089 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.088+0000 7fc776ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:09:47.089 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.088+0000 7fc776ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:47.089 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.088+0000 7fc776ffd640 1 -- 172.21.15.156:0/3403554152 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc770001a10 con 0x7fc7780ae280 2024-03-25T18:09:47.089 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.088+0000 7fc776ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:47.090 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.088+0000 7fc776ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:47.090 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.088+0000 7fc776ffd640 10 monclient: dump: 2024-03-25T18:09:47.090 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:47.090 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:47.090 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:47.090 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:47.090 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:47.090 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:47.090 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:47.090 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:47.090 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:47.090 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:47.090 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.088+0000 7fc77e3cb640 5 monclient: authenticate success, global_id 4287 2024-03-25T18:09:47.090 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.088+0000 7fc77e3cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:47.106 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.105+0000 7fc776ffd640 1 -- 172.21.15.156:0/3403554152 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc770001b70 con 0x7fc7780ae280 2024-03-25T18:09:47.107 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.105+0000 7fc776ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:47.107 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.105+0000 7fc776ffd640 1 -- 172.21.15.156:0/3403554152 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc770001f80 con 0x7fc7780ae280 2024-03-25T18:09:47.107 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.105+0000 7fc77e3cb640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:47.107 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.105+0000 7fc77e3cb640 4 set_mon_vals no callback set 2024-03-25T18:09:47.107 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.106+0000 7fc776ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:47.107 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.106+0000 7fc776ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:47.107 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.106+0000 7fc776ffd640 10 monclient: dump: 2024-03-25T18:09:47.107 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:47.107 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:47.107 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:47.107 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:47.107 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:47.107 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:47.107 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:47.107 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:47.108 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:47.108 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:47.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.106+0000 7fc77e3cb640 10 monclient: shutdown 2024-03-25T18:09:47.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.106+0000 7fc77e3cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:47.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.106+0000 7fc77e3cb640 1 -- 172.21.15.156:0/3403554152 >> v1:172.21.15.156:6789/0 conn(0x7fc7780ae280 legacy=0x7fc778145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:47.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.106+0000 7fc77e3cb640 1 -- 172.21.15.156:0/3403554152 shutdown_connections 2024-03-25T18:09:47.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.106+0000 7fc77e3cb640 1 -- 172.21.15.156:0/3403554152 >> 172.21.15.156:0/3403554152 conn(0x7fc7780b0ef0 msgr2=0x7fc7780b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:47.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.106+0000 7fc77e3cb640 1 -- 172.21.15.156:0/3403554152 shutdown_connections 2024-03-25T18:09:47.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.107+0000 7fc77e3cb640 1 -- 172.21.15.156:0/3403554152 wait complete. 2024-03-25T18:09:47.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.107+0000 7fc77e3cb640 10 monclient: build_initial_monmap 2024-03-25T18:09:47.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.107+0000 7fc77e3cb640 10 monclient: monmap: 2024-03-25T18:09:47.110 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:09:47.110 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:47.110 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:09:47.110 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:09:47.111 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:47.111 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:09:47.111 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:09:47.111 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:09:47.111 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:09:47.111 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:47.111 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.108+0000 7fc77e3cb640 1 librados: starting msgr at 2024-03-25T18:09:47.111 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.108+0000 7fc77e3cb640 1 librados: starting objecter 2024-03-25T18:09:47.111 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.108+0000 7fc77e3cb640 1 Processor -- start 2024-03-25T18:09:47.111 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.108+0000 7fc77e3cb640 1 -- start start 2024-03-25T18:09:47.111 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.108+0000 7fc77e3cb640 1 librados: setting wanted keys 2024-03-25T18:09:47.111 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.108+0000 7fc77e3cb640 1 librados: calling monclient init 2024-03-25T18:09:47.111 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.108+0000 7fc77e3cb640 10 monclient: init 2024-03-25T18:09:47.111 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.109+0000 7fc77e3cb640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:47.111 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.109+0000 7fc77e3cb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:09:47.111 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.109+0000 7fc77e3cb640 10 monclient(hunting): picked mon.noname-c con 0x7fc778061890 addr v1:172.21.15.156:6790/0 2024-03-25T18:09:47.111 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.109+0000 7fc77e3cb640 10 monclient(hunting): picked mon.noname-a con 0x7fc7780ae280 addr v1:172.21.15.156:6789/0 2024-03-25T18:09:47.112 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.109+0000 7fc77e3cb640 10 monclient(hunting): picked mon.noname-b con 0x7fc778139050 addr v1:172.21.15.194:6789/0 2024-03-25T18:09:47.112 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.109+0000 7fc77e3cb640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc778145bb0 con 0x7fc7780ae280 2024-03-25T18:09:47.112 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.109+0000 7fc77e3cb640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc77813a610 con 0x7fc778139050 2024-03-25T18:09:47.112 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.109+0000 7fc77e3cb640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc77813a8f0 con 0x7fc778061890 2024-03-25T18:09:47.112 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.109+0000 7fc77e3cb640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:47.112 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.109+0000 7fc77e3cb640 10 monclient(hunting): authenticate will time out at 7068.420157s 2024-03-25T18:09:47.128 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.127+0000 7fc777fff640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7fc778061890 0x7fc77813b2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:37994/0 (socket says 172.21.15.156:37994) 2024-03-25T18:09:47.128 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.127+0000 7fc777fff640 1 -- 172.21.15.156:0/741862353 learned_addr learned my addr 172.21.15.156:0/741862353 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:09:47.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.128+0000 7fc774ff9640 10 client.?.objecter ms_handle_connect 0x7fc778061890 2024-03-25T18:09:47.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.128+0000 7fc774ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:47.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.131+0000 7fc774ff9640 10 client.?.objecter ms_handle_connect 0x7fc778139050 2024-03-25T18:09:47.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.131+0000 7fc774ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:47.139 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.139+0000 7fc774ff9640 1 -- 172.21.15.156:0/741862353 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 125377277 0 0) 0x7fc77813a8f0 con 0x7fc778061890 2024-03-25T18:09:47.139 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.139+0000 7fc774ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:09:47.139 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.139+0000 7fc774ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:47.140 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.139+0000 7fc774ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:47.140 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.139+0000 7fc774ff9640 10 monclient(hunting): my global_id is 4298 2024-03-25T18:09:47.140 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.139+0000 7fc774ff9640 1 -- 172.21.15.156:0/741862353 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc764004eb0 con 0x7fc778061890 2024-03-25T18:09:47.142 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.141+0000 7fc774ff9640 1 -- 172.21.15.156:0/741862353 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 807369040 0 0) 0x7fc77813a610 con 0x7fc778139050 2024-03-25T18:09:47.142 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.141+0000 7fc774ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:09:47.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.141+0000 7fc774ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:47.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.141+0000 7fc774ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:47.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.141+0000 7fc774ff9640 10 monclient(hunting): my global_id is 4288 2024-03-25T18:09:47.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.142+0000 7fc774ff9640 1 -- 172.21.15.156:0/741862353 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc77813a8f0 con 0x7fc778139050 2024-03-25T18:09:47.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.144+0000 7fc774ff9640 10 client.?.objecter ms_handle_connect 0x7fc7780ae280 2024-03-25T18:09:47.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.144+0000 7fc774ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:47.150 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.149+0000 7fc774ff9640 1 -- 172.21.15.156:0/741862353 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1838834254 0 0) 0x7fc764004eb0 con 0x7fc778061890 2024-03-25T18:09:47.150 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.149+0000 7fc774ff9640 1 -- 172.21.15.156:0/741862353 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc77813a610 con 0x7fc778061890 2024-03-25T18:09:47.150 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.149+0000 7fc774ff9640 1 -- 172.21.15.156:0/741862353 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc7680019f0 con 0x7fc778061890 2024-03-25T18:09:47.150 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.149+0000 7fc774ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:47.150 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.149+0000 7fc774ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:47.150 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.149+0000 7fc774ff9640 10 monclient(hunting): dump: 2024-03-25T18:09:47.151 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:47.151 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:47.151 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:47.151 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:47.151 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:47.151 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:47.151 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:47.151 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:47.151 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:47.151 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:47.151 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.149+0000 7fc774ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:47.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.161+0000 7fc774ff9640 1 -- 172.21.15.156:0/741862353 <== mon.2 v1:172.21.15.156:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1322748263 0 0) 0x7fc77813a610 con 0x7fc778061890 2024-03-25T18:09:47.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.161+0000 7fc774ff9640 1 -- 172.21.15.156:0/741862353 >> v1:172.21.15.194:6789/0 conn(0x7fc778139050 legacy=0x7fc7781e2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:47.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.161+0000 7fc774ff9640 1 -- 172.21.15.156:0/741862353 >> v1:172.21.15.156:6789/0 conn(0x7fc7780ae280 legacy=0x7fc778146090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:47.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.161+0000 7fc774ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:09:47.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.161+0000 7fc774ff9640 1 monclient: found mon.c 2024-03-25T18:09:47.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.161+0000 7fc774ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:47.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.161+0000 7fc774ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:09:47.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.161+0000 7fc774ff9640 1 -- 172.21.15.156:0/741862353 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc77813abd0 con 0x7fc778061890 2024-03-25T18:09:47.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.161+0000 7fc774ff9640 10 monclient: _finish_auth 0 2024-03-25T18:09:47.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.161+0000 7fc774ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:09:47.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.161+0000 7fc774ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:47.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.161+0000 7fc77e3cb640 5 monclient: authenticate success, global_id 4298 2024-03-25T18:09:47.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.161+0000 7fc77e3cb640 10 monclient: _renew_subs 2024-03-25T18:09:47.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.161+0000 7fc77e3cb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:09:47.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.161+0000 7fc77e3cb640 1 -- 172.21.15.156:0/741862353 --> v1:172.21.15.156:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc77813aeb0 con 0x7fc778061890 2024-03-25T18:09:47.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.161+0000 7fc77e3cb640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:47.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.161+0000 7fc77e3cb640 10 monclient: _renew_subs 2024-03-25T18:09:47.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.161+0000 7fc77e3cb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:09:47.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.162+0000 7fc77e3cb640 1 -- 172.21.15.156:0/741862353 --> v1:172.21.15.156:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc778139ea0 con 0x7fc778061890 2024-03-25T18:09:47.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.162+0000 7fc77e3cb640 1 librados: init done 2024-03-25T18:09:47.164 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.162+0000 7fc77e3cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:47.164 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.162+0000 7fc77e3cb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:47.164 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.162+0000 7fc77e3cb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:09:47.164 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.162+0000 7fc77e3cb640 1 -- 172.21.15.156:0/741862353 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7780627b0 con 0x7fc778061890 2024-03-25T18:09:47.477 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.476+0000 7fc774ff9640 1 -- 172.21.15.156:0/741862353 <== mon.2 v1:172.21.15.156:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc768001c50 con 0x7fc778061890 2024-03-25T18:09:47.477 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.476+0000 7fc774ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:47.477 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.476+0000 7fc774ff9640 1 -- 172.21.15.156:0/741862353 <== mon.2 v1:172.21.15.156:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc768004c30 con 0x7fc778061890 2024-03-25T18:09:47.477 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.476+0000 7fc774ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:47.477 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.476+0000 7fc774ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:47.477 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.476+0000 7fc774ff9640 10 monclient: dump: 2024-03-25T18:09:47.477 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:47.477 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:47.477 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:47.477 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:47.477 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:47.477 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:47.477 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:47.478 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:47.478 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:47.478 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:47.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.476+0000 7fc7767fc640 4 set_mon_vals no callback set 2024-03-25T18:09:47.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.477+0000 7fc774ff9640 1 -- 172.21.15.156:0/741862353 <== mon.2 v1:172.21.15.156:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7fc76801ac00 con 0x7fc778061890 2024-03-25T18:09:47.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.477+0000 7fc7777fe640 10 monclient: get_auth_request con 0x7fc76406dd10 auth_method 0 2024-03-25T18:09:47.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.478+0000 7fc774ff9640 10 client.4298.objecter ms_handle_connect 0x7fc76406dd10 2024-03-25T18:09:47.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.482+0000 7fc774ff9640 1 -- 172.21.15.156:0/741862353 <== mon.2 v1:172.21.15.156:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7fc7680548f0 con 0x7fc778061890 2024-03-25T18:09:47.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.482+0000 7fc774ff9640 10 client.4298.objecter ms_dispatch 0x7fc778000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:47.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.482+0000 7fc774ff9640 3 client.4298.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:47.484 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.482+0000 7fc774ff9640 3 client.4298.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:47.484 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.482+0000 7fc774ff9640 20 client.4298.objecter dump_active .. 0 homeless 2024-03-25T18:09:47.484 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.482+0000 7fc774ff9640 1 -- 172.21.15.156:0/741862353 <== mon.2 v1:172.21.15.156:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc768050230 con 0x7fc778061890 2024-03-25T18:09:47.484 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.482+0000 7fc774ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:47.484 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.482+0000 7fc774ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:47.736 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.735+0000 7fc77e3cb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:47.736 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.736+0000 7fc77e3cb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:47.737 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.736+0000 7fc77e3cb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:09:47.737 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.736+0000 7fc77e3cb640 1 -- 172.21.15.156:0/741862353 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc778069f40 con 0x7fc778061890 2024-03-25T18:09:47.766 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.765+0000 7fc774ff9640 1 -- 172.21.15.156:0/741862353 <== mon.2 v1:172.21.15.156:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7fc768052330 con 0x7fc778061890 2024-03-25T18:09:47.766 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.765+0000 7fc774ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:47.766 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.765+0000 7fc774ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:47.766 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:09:47.766 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:47.769 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.767+0000 7fc77e3cb640 10 librados: watch_flush enter 2024-03-25T18:09:47.769 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.767+0000 7fc77e3cb640 10 librados: watch_flush exit 2024-03-25T18:09:47.769 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.767+0000 7fc77e3cb640 20 client.4298.objecter shutdown clearing up homeless session... 2024-03-25T18:09:47.769 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.767+0000 7fc77e3cb640 10 client.4298.objecter successfully canceled tick 2024-03-25T18:09:47.769 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.768+0000 7fc77e3cb640 1 -- 172.21.15.156:0/741862353 >> v1:172.21.15.194:6800/2946427713 conn(0x7fc76406dd10 legacy=0x7fc7640701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:47.769 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.768+0000 7fc77e3cb640 10 monclient: shutdown 2024-03-25T18:09:47.769 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.768+0000 7fc77e3cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:47.769 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.768+0000 7fc77e3cb640 1 -- 172.21.15.156:0/741862353 >> v1:172.21.15.156:6790/0 conn(0x7fc778061890 legacy=0x7fc77813b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:47.769 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.768+0000 7fc77e3cb640 1 -- 172.21.15.156:0/741862353 shutdown_connections 2024-03-25T18:09:47.769 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.768+0000 7fc77e3cb640 1 -- 172.21.15.156:0/741862353 >> 172.21.15.156:0/741862353 conn(0x7fc7780b0ef0 msgr2=0x7fc77813f470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:47.769 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.768+0000 7fc77e3cb640 1 -- 172.21.15.156:0/741862353 shutdown_connections 2024-03-25T18:09:47.769 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.768+0000 7fc77e3cb640 1 -- 172.21.15.156:0/741862353 wait complete. 2024-03-25T18:09:47.769 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:47.768+0000 7fc77e3cb640 1 librados: shutdown 2024-03-25T18:09:47.782 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:53.141 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:01:59.131 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:53.148 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:01:59.138 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:53.180 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:01:59.171 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:53.191 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:01:59.158 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:53.204 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:01:59.171 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:53.206 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:01:59.197 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:53.219 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:01:59.210 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:53.235 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:01:59.202 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:53.246 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:01:59.237 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:54.589 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:54.706 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.705+0000 7f929cc93640 10 monclient: get_monmap_and_config 2024-03-25T18:09:54.706 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.705+0000 7f929cc93640 10 monclient: build_initial_monmap 2024-03-25T18:09:54.706 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.705+0000 7f929cc93640 10 monclient: monmap: 2024-03-25T18:09:54.706 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:09:54.706 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:54.706 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:09:54.706531+0000 2024-03-25T18:09:54.706 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:09:54.706531+0000 2024-03-25T18:09:54.706 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:54.707 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:54.707 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:09:54.707 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:09:54.707 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:09:54.707 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:54.708 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.706+0000 7f929cc93640 1 Processor -- start 2024-03-25T18:09:54.708 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.706+0000 7f929cc93640 1 -- start start 2024-03-25T18:09:54.708 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.706+0000 7f929cc93640 10 monclient: init 2024-03-25T18:09:54.708 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.707+0000 7f929cc93640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:54.708 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.707+0000 7f929cc93640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:54.708 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.707+0000 7f929cc93640 10 monclient(hunting): picked mon.noname-c con 0x7f92980a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:09:54.708 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.707+0000 7f929cc93640 10 monclient(hunting): picked mon.noname-b con 0x7f929813f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:09:54.708 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.707+0000 7f929cc93640 10 monclient(hunting): picked mon.noname-a con 0x7f92981430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:09:54.708 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.707+0000 7f929cc93640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9298142520 con 0x7f92981430c0 2024-03-25T18:09:54.708 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.707+0000 7f929cc93640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92980b28a0 con 0x7f92980a98f0 2024-03-25T18:09:54.709 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.707+0000 7f929cc93640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92980b2a30 con 0x7f929813f710 2024-03-25T18:09:54.709 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.707+0000 7f929cc93640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:54.709 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.707+0000 7f929cc93640 10 monclient(hunting): authenticate will time out at 7076.018062s 2024-03-25T18:09:54.720 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.719+0000 7f928ffff640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f929813f710 0x7f92981429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:56676/0 (socket says 172.21.15.156:56676) 2024-03-25T18:09:54.720 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.719+0000 7f928ffff640 1 -- 172.21.15.156:0/787448846 learned_addr learned my addr 172.21.15.156:0/787448846 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:09:54.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.731+0000 7f9295d9b640 1 -- 172.21.15.156:0/787448846 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776016944 0 0) 0x7f92980b2a30 con 0x7f929813f710 2024-03-25T18:09:54.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.731+0000 7f9295d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:09:54.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.731+0000 7f9295d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:54.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.731+0000 7f9295d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:54.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.731+0000 7f9295d9b640 10 monclient(hunting): my global_id is 4301 2024-03-25T18:09:54.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.731+0000 7f9295d9b640 1 -- 172.21.15.156:0/787448846 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9284003520 con 0x7f929813f710 2024-03-25T18:09:54.738 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.738+0000 7f9295d9b640 1 -- 172.21.15.156:0/787448846 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1623506566 0 0) 0x7f92980b28a0 con 0x7f92980a98f0 2024-03-25T18:09:54.738 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.738+0000 7f9295d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:09:54.739 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.738+0000 7f9295d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:54.739 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.738+0000 7f9295d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:54.739 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.738+0000 7f9295d9b640 10 monclient(hunting): my global_id is 4291 2024-03-25T18:09:54.739 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.738+0000 7f9295d9b640 1 -- 172.21.15.156:0/787448846 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92980b2a30 con 0x7f92980a98f0 2024-03-25T18:09:54.746 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.746+0000 7f9295d9b640 1 -- 172.21.15.156:0/787448846 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2128351338 0 0) 0x7f9298142520 con 0x7f92981430c0 2024-03-25T18:09:54.746 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.746+0000 7f9295d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:09:54.746 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.746+0000 7f9295d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:54.746 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.746+0000 7f9295d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:54.747 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.746+0000 7f9295d9b640 10 monclient(hunting): my global_id is 4293 2024-03-25T18:09:54.747 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.746+0000 7f9295d9b640 1 -- 172.21.15.156:0/787448846 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92980b28a0 con 0x7f92981430c0 2024-03-25T18:09:54.761 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.760+0000 7f9295d9b640 1 -- 172.21.15.156:0/787448846 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3677398955 0 0) 0x7f9284003520 con 0x7f929813f710 2024-03-25T18:09:54.761 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.760+0000 7f9295d9b640 1 -- 172.21.15.156:0/787448846 >> v1:172.21.15.194:6789/0 conn(0x7f92980a98f0 legacy=0x7f92980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:54.761 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.760+0000 7f9295d9b640 1 -- 172.21.15.156:0/787448846 >> v1:172.21.15.156:6789/0 conn(0x7f92981430c0 legacy=0x7f929814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:54.761 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.760+0000 7f9295d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:09:54.761 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.760+0000 7f9295d9b640 1 monclient: found mon.noname-b 2024-03-25T18:09:54.761 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.760+0000 7f9295d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:54.761 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.760+0000 7f9295d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.156:6790/0 2024-03-25T18:09:54.762 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.760+0000 7f9295d9b640 1 -- 172.21.15.156:0/787448846 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92980b44c0 con 0x7f929813f710 2024-03-25T18:09:54.762 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.760+0000 7f9295d9b640 10 monclient: _finish_auth 0 2024-03-25T18:09:54.762 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.760+0000 7f9295d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:09:54.762 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.760+0000 7f9295d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:54.762 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.760+0000 7f9295d9b640 1 -- 172.21.15.156:0/787448846 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9280001690 con 0x7f929813f710 2024-03-25T18:09:54.762 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.760+0000 7f9295d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:54.762 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.760+0000 7f9295d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:54.762 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.760+0000 7f9295d9b640 10 monclient: dump: 2024-03-25T18:09:54.762 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:54.762 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:54.762 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:54.762 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:54.762 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:54.762 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:54.763 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:54.763 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:54.763 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:54.763 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:54.763 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.760+0000 7f929cc93640 5 monclient: authenticate success, global_id 4301 2024-03-25T18:09:54.763 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.760+0000 7f929cc93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:54.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.780+0000 7f9295d9b640 1 -- 172.21.15.156:0/787448846 <== mon.2 v1:172.21.15.156:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92800017f0 con 0x7f929813f710 2024-03-25T18:09:54.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.780+0000 7f9295d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:54.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.780+0000 7f9295d9b640 1 -- 172.21.15.156:0/787448846 <== mon.2 v1:172.21.15.156:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9280001c00 con 0x7f929813f710 2024-03-25T18:09:54.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.780+0000 7f9295d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:54.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.780+0000 7f9295d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:54.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.780+0000 7f9295d9b640 10 monclient: dump: 2024-03-25T18:09:54.781 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:54.781 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:54.781 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:54.781 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:54.781 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:54.781 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:54.781 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:54.781 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:54.782 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:54.782 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:54.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.780+0000 7f929cc93640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:54.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.780+0000 7f929cc93640 4 set_mon_vals no callback set 2024-03-25T18:09:54.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.781+0000 7f929cc93640 10 monclient: shutdown 2024-03-25T18:09:54.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.781+0000 7f929cc93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:54.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.781+0000 7f929cc93640 1 -- 172.21.15.156:0/787448846 >> v1:172.21.15.156:6790/0 conn(0x7f929813f710 legacy=0x7f92981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:54.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.781+0000 7f929cc93640 1 -- 172.21.15.156:0/787448846 shutdown_connections 2024-03-25T18:09:54.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.781+0000 7f929cc93640 1 -- 172.21.15.156:0/787448846 >> 172.21.15.156:0/787448846 conn(0x7f92980b7ea0 msgr2=0x7f92980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:54.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.781+0000 7f929cc93640 1 -- 172.21.15.156:0/787448846 shutdown_connections 2024-03-25T18:09:54.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.782+0000 7f929cc93640 1 -- 172.21.15.156:0/787448846 wait complete. 2024-03-25T18:09:54.785 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.782+0000 7f929cc93640 10 monclient: build_initial_monmap 2024-03-25T18:09:54.785 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.782+0000 7f929cc93640 10 monclient: monmap: 2024-03-25T18:09:54.785 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:09:54.785 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:54.785 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:09:54.785 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:09:54.785 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:54.785 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:09:54.785 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:09:54.785 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:09:54.785 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:09:54.786 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:54.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.783+0000 7f929cc93640 1 librados: starting msgr at 2024-03-25T18:09:54.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.783+0000 7f929cc93640 1 librados: starting objecter 2024-03-25T18:09:54.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.783+0000 7f929cc93640 1 Processor -- start 2024-03-25T18:09:54.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.783+0000 7f929cc93640 1 -- start start 2024-03-25T18:09:54.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.783+0000 7f929cc93640 1 librados: setting wanted keys 2024-03-25T18:09:54.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.783+0000 7f929cc93640 1 librados: calling monclient init 2024-03-25T18:09:54.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.783+0000 7f929cc93640 10 monclient: init 2024-03-25T18:09:54.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.784+0000 7f929cc93640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:54.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.784+0000 7f929cc93640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:54.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.784+0000 7f929cc93640 10 monclient(hunting): picked mon.noname-b con 0x7f92980a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:09:54.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.784+0000 7f929cc93640 10 monclient(hunting): picked mon.noname-c con 0x7f929813f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:09:54.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.784+0000 7f929cc93640 10 monclient(hunting): picked mon.noname-a con 0x7f92981430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:09:54.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.784+0000 7f929cc93640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92980abc80 con 0x7f92981430c0 2024-03-25T18:09:54.787 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.784+0000 7f929cc93640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92981d7780 con 0x7f92980a98f0 2024-03-25T18:09:54.787 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.784+0000 7f929cc93640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92981399a0 con 0x7f929813f710 2024-03-25T18:09:54.787 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.784+0000 7f929cc93640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:54.787 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.784+0000 7f929cc93640 10 monclient(hunting): authenticate will time out at 7076.094944s 2024-03-25T18:09:54.802 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.801+0000 7f928ffff640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f929813f710 0x7f92981d4840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:56740/0 (socket says 172.21.15.156:56740) 2024-03-25T18:09:54.802 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.801+0000 7f928ffff640 1 -- 172.21.15.156:0/3184130823 learned_addr learned my addr 172.21.15.156:0/3184130823 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:09:54.805 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.805+0000 7f928effd640 10 client.?.objecter ms_handle_connect 0x7f929813f710 2024-03-25T18:09:54.805 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.805+0000 7f928effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:54.806 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.805+0000 7f928effd640 10 client.?.objecter ms_handle_connect 0x7f92980a98f0 2024-03-25T18:09:54.806 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.805+0000 7f928effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:54.814 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.813+0000 7f928effd640 1 -- 172.21.15.156:0/3184130823 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3857322238 0 0) 0x7f92981399a0 con 0x7f929813f710 2024-03-25T18:09:54.814 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.813+0000 7f928effd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:54.814 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.813+0000 7f928effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:54.814 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.813+0000 7f928effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:54.814 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.813+0000 7f928effd640 10 monclient(hunting): my global_id is 4304 2024-03-25T18:09:54.814 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.813+0000 7f928effd640 1 -- 172.21.15.156:0/3184130823 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9278004eb0 con 0x7f929813f710 2024-03-25T18:09:54.815 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.815+0000 7f928effd640 10 client.?.objecter ms_handle_connect 0x7f92981430c0 2024-03-25T18:09:54.815 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.815+0000 7f928effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:54.818 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.817+0000 7f928effd640 1 -- 172.21.15.156:0/3184130823 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4081916074 0 0) 0x7f92981d7780 con 0x7f92980a98f0 2024-03-25T18:09:54.818 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.817+0000 7f928effd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:54.818 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.817+0000 7f928effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:54.818 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.817+0000 7f928effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:54.818 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.817+0000 7f928effd640 10 monclient(hunting): my global_id is 4294 2024-03-25T18:09:54.818 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.817+0000 7f928effd640 1 -- 172.21.15.156:0/3184130823 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92981399a0 con 0x7f92980a98f0 2024-03-25T18:09:54.828 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.827+0000 7f928effd640 1 -- 172.21.15.156:0/3184130823 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3086918769 0 0) 0x7f92981399a0 con 0x7f92980a98f0 2024-03-25T18:09:54.828 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.827+0000 7f928effd640 1 -- 172.21.15.156:0/3184130823 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92981d7780 con 0x7f92980a98f0 2024-03-25T18:09:54.828 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.827+0000 7f928effd640 1 -- 172.21.15.156:0/3184130823 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9288001970 con 0x7f92980a98f0 2024-03-25T18:09:54.828 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.827+0000 7f928effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:54.828 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.827+0000 7f928effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:54.828 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.827+0000 7f928effd640 10 monclient(hunting): dump: 2024-03-25T18:09:54.828 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:54.828 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:54.828 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:54.829 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:54.829 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:54.829 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:54.829 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:54.829 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:54.829 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:54.829 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:54.829 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.827+0000 7f928effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:54.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.835+0000 7f928effd640 1 -- 172.21.15.156:0/3184130823 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3382912352 0 0) 0x7f92980abc80 con 0x7f92981430c0 2024-03-25T18:09:54.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.835+0000 7f928effd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:54.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.835+0000 7f928effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:54.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.835+0000 7f928effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:54.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.835+0000 7f928effd640 10 monclient(hunting): my global_id is 4296 2024-03-25T18:09:54.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.835+0000 7f928effd640 1 -- 172.21.15.156:0/3184130823 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92981399a0 con 0x7f92981430c0 2024-03-25T18:09:54.839 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.838+0000 7f928effd640 1 -- 172.21.15.156:0/3184130823 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3008542345 0 0) 0x7f92981d7780 con 0x7f92980a98f0 2024-03-25T18:09:54.839 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.838+0000 7f928effd640 1 -- 172.21.15.156:0/3184130823 >> v1:172.21.15.156:6790/0 conn(0x7f929813f710 legacy=0x7f92981d4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:54.840 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.838+0000 7f928effd640 1 -- 172.21.15.156:0/3184130823 >> v1:172.21.15.156:6789/0 conn(0x7f92981430c0 legacy=0x7f9298138600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:54.840 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.838+0000 7f928effd640 10 monclient: _finish_hunting 0 2024-03-25T18:09:54.840 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.838+0000 7f928effd640 1 monclient: found mon.b 2024-03-25T18:09:54.840 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.838+0000 7f928effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:54.840 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.838+0000 7f928effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:09:54.840 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.838+0000 7f928effd640 1 -- 172.21.15.156:0/3184130823 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92981d41e0 con 0x7f92980a98f0 2024-03-25T18:09:54.840 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.838+0000 7f928effd640 10 monclient: _finish_auth 0 2024-03-25T18:09:54.840 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.838+0000 7f928effd640 10 monclient: _check_auth_tickets 2024-03-25T18:09:54.840 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.838+0000 7f928effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:54.840 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.838+0000 7f929cc93640 5 monclient: authenticate success, global_id 4294 2024-03-25T18:09:54.840 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.838+0000 7f929cc93640 10 monclient: _renew_subs 2024-03-25T18:09:54.840 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.838+0000 7f929cc93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:09:54.840 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.838+0000 7f929cc93640 1 -- 172.21.15.156:0/3184130823 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92981d8370 con 0x7f92980a98f0 2024-03-25T18:09:54.840 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.838+0000 7f929cc93640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:54.841 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.838+0000 7f929cc93640 10 monclient: _renew_subs 2024-03-25T18:09:54.841 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.838+0000 7f929cc93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:09:54.841 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.838+0000 7f929cc93640 1 -- 172.21.15.156:0/3184130823 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92981d5130 con 0x7f92980a98f0 2024-03-25T18:09:54.841 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.838+0000 7f929cc93640 1 librados: init done 2024-03-25T18:09:54.841 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.839+0000 7f929cc93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:54.841 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.839+0000 7f929cc93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:54.841 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.839+0000 7f929cc93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:09:54.842 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:54.839+0000 7f929cc93640 1 -- 172.21.15.156:0/3184130823 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92980627b0 con 0x7f92980a98f0 2024-03-25T18:09:55.119 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.119+0000 7f928effd640 1 -- 172.21.15.156:0/3184130823 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9288003610 con 0x7f92980a98f0 2024-03-25T18:09:55.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.119+0000 7f928effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:55.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.119+0000 7f928effd640 1 -- 172.21.15.156:0/3184130823 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9288003920 con 0x7f92980a98f0 2024-03-25T18:09:55.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.119+0000 7f928effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:55.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.119+0000 7f928effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:55.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.119+0000 7f928effd640 10 monclient: dump: 2024-03-25T18:09:55.120 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:09:55.120 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:09:55.120 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:55.120 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:09:55.120 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:55.120 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:09:55.120 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:09:55.120 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:09:55.120 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:09:55.121 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:09:55.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.119+0000 7f929559a640 4 set_mon_vals no callback set 2024-03-25T18:09:55.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.119+0000 7f928effd640 1 -- 172.21.15.156:0/3184130823 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f92880198f0 con 0x7f92980a98f0 2024-03-25T18:09:55.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.120+0000 7f928ffff640 10 monclient: get_auth_request con 0x7f927806ddc0 auth_method 0 2024-03-25T18:09:55.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.120+0000 7f928effd640 10 client.4294.objecter ms_handle_connect 0x7f927806ddc0 2024-03-25T18:09:55.125 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.124+0000 7f928effd640 1 -- 172.21.15.156:0/3184130823 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f9288053580 con 0x7f92980a98f0 2024-03-25T18:09:55.125 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.124+0000 7f928effd640 10 client.4294.objecter ms_dispatch 0x7f9298000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:55.126 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.124+0000 7f928effd640 3 client.4294.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:55.126 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.124+0000 7f928effd640 3 client.4294.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:55.126 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.125+0000 7f928effd640 20 client.4294.objecter dump_active .. 0 homeless 2024-03-25T18:09:55.126 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.125+0000 7f928effd640 1 -- 172.21.15.156:0/3184130823 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f928804eec0 con 0x7f92980a98f0 2024-03-25T18:09:55.126 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.125+0000 7f928effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:55.126 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.125+0000 7f928effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:55.380 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.379+0000 7f929cc93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:55.380 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.379+0000 7f929cc93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:55.380 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.379+0000 7f929cc93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:09:55.380 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.379+0000 7f929cc93640 1 -- 172.21.15.156:0/3184130823 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92981d4370 con 0x7f92980a98f0 2024-03-25T18:09:55.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.407+0000 7f928effd640 1 -- 172.21.15.156:0/3184130823 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f9288050fc0 con 0x7f92980a98f0 2024-03-25T18:09:55.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.407+0000 7f928effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:55.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.407+0000 7f928effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:55.408 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:09:55.408 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:55.410 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.409+0000 7f929cc93640 10 librados: watch_flush enter 2024-03-25T18:09:55.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.409+0000 7f929cc93640 10 librados: watch_flush exit 2024-03-25T18:09:55.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.409+0000 7f929cc93640 20 client.4294.objecter shutdown clearing up homeless session... 2024-03-25T18:09:55.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.409+0000 7f929cc93640 10 client.4294.objecter successfully canceled tick 2024-03-25T18:09:55.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.409+0000 7f929cc93640 1 -- 172.21.15.156:0/3184130823 >> v1:172.21.15.194:6800/2946427713 conn(0x7f927806ddc0 legacy=0x7f9278070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:55.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.409+0000 7f929cc93640 10 monclient: shutdown 2024-03-25T18:09:55.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.409+0000 7f929cc93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:55.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.409+0000 7f929cc93640 1 -- 172.21.15.156:0/3184130823 >> v1:172.21.15.194:6789/0 conn(0x7f92980a98f0 legacy=0x7f92981d7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:55.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.409+0000 7f929cc93640 1 -- 172.21.15.156:0/3184130823 shutdown_connections 2024-03-25T18:09:55.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.409+0000 7f929cc93640 1 -- 172.21.15.156:0/3184130823 >> 172.21.15.156:0/3184130823 conn(0x7f92980b7ea0 msgr2=0x7f9298074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:55.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.409+0000 7f929cc93640 1 -- 172.21.15.156:0/3184130823 shutdown_connections 2024-03-25T18:09:55.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.409+0000 7f929cc93640 1 -- 172.21.15.156:0/3184130823 wait complete. 2024-03-25T18:09:55.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:09:55.410+0000 7f929cc93640 1 librados: shutdown 2024-03-25T18:09:55.422 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:02.229 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:02.333 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.332+0000 7f6178e2e640 10 monclient: get_monmap_and_config 2024-03-25T18:10:02.333 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.332+0000 7f6178e2e640 10 monclient: build_initial_monmap 2024-03-25T18:10:02.333 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.332+0000 7f6178e2e640 10 monclient: monmap: 2024-03-25T18:10:02.333 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:10:02.334 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:02.334 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:10:02.333885+0000 2024-03-25T18:10:02.334 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:10:02.333885+0000 2024-03-25T18:10:02.334 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:02.334 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:02.334 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:10:02.334 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:10:02.334 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:10:02.334 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:02.335 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.333+0000 7f6178e2e640 1 Processor -- start 2024-03-25T18:10:02.335 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.333+0000 7f6178e2e640 1 -- start start 2024-03-25T18:10:02.335 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.333+0000 7f6178e2e640 10 monclient: init 2024-03-25T18:10:02.336 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.334+0000 7f6178e2e640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:02.336 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.334+0000 7f6178e2e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:10:02.336 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.334+0000 7f6178e2e640 10 monclient(hunting): picked mon.noname-c con 0x7f61740a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:10:02.336 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.334+0000 7f6178e2e640 10 monclient(hunting): picked mon.noname-b con 0x7f617413f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:10:02.336 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.334+0000 7f6178e2e640 10 monclient(hunting): picked mon.noname-a con 0x7f61741430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:10:02.336 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.334+0000 7f6178e2e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6174142520 con 0x7f61741430c0 2024-03-25T18:10:02.336 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.334+0000 7f6178e2e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61740b28a0 con 0x7f61740a98f0 2024-03-25T18:10:02.336 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.334+0000 7f6178e2e640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61740b2a30 con 0x7f617413f710 2024-03-25T18:10:02.336 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.334+0000 7f6178e2e640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:02.336 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.334+0000 7f6178e2e640 10 monclient(hunting): authenticate will time out at 7083.645411s 2024-03-25T18:10:02.349 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.348+0000 7f6172d9d640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f61741430c0 0x7f617414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:48632/0 (socket says 172.21.15.156:48632) 2024-03-25T18:10:02.349 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.348+0000 7f6172d9d640 1 -- 172.21.15.156:0/2258992805 learned_addr learned my addr 172.21.15.156:0/2258992805 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:10:02.368 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.368+0000 7f617159a640 1 -- 172.21.15.156:0/2258992805 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1017603320 0 0) 0x7f61740b2a30 con 0x7f617413f710 2024-03-25T18:10:02.368 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.368+0000 7f617159a640 10 monclient(hunting): _negotiate 2024-03-25T18:10:02.368 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.368+0000 7f617159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:02.368 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.368+0000 7f617159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:02.369 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.368+0000 7f617159a640 10 monclient(hunting): my global_id is 4307 2024-03-25T18:10:02.369 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.368+0000 7f617159a640 1 -- 172.21.15.156:0/2258992805 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f615c003520 con 0x7f617413f710 2024-03-25T18:10:02.372 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.371+0000 7f617159a640 1 -- 172.21.15.156:0/2258992805 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2168708183 0 0) 0x7f61740b28a0 con 0x7f61740a98f0 2024-03-25T18:10:02.372 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.371+0000 7f617159a640 10 monclient(hunting): _negotiate 2024-03-25T18:10:02.372 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.371+0000 7f617159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:02.372 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.371+0000 7f617159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:02.372 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.371+0000 7f617159a640 10 monclient(hunting): my global_id is 4297 2024-03-25T18:10:02.373 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.371+0000 7f617159a640 1 -- 172.21.15.156:0/2258992805 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61740b2a30 con 0x7f61740a98f0 2024-03-25T18:10:02.384 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.383+0000 7f617159a640 1 -- 172.21.15.156:0/2258992805 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2477810802 0 0) 0x7f61740b2a30 con 0x7f61740a98f0 2024-03-25T18:10:02.384 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.383+0000 7f617159a640 1 -- 172.21.15.156:0/2258992805 >> v1:172.21.15.156:6790/0 conn(0x7f617413f710 legacy=0x7f61741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:02.384 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.383+0000 7f617159a640 1 -- 172.21.15.156:0/2258992805 >> v1:172.21.15.156:6789/0 conn(0x7f61741430c0 legacy=0x7f617414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:02.384 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.383+0000 7f617159a640 10 monclient: _finish_hunting 0 2024-03-25T18:10:02.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.383+0000 7f617159a640 1 monclient: found mon.noname-c 2024-03-25T18:10:02.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.383+0000 7f617159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:02.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.383+0000 7f617159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:10:02.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.383+0000 7f617159a640 1 -- 172.21.15.156:0/2258992805 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61740b44c0 con 0x7f61740a98f0 2024-03-25T18:10:02.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.383+0000 7f617159a640 10 monclient: _finish_auth 0 2024-03-25T18:10:02.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.383+0000 7f617159a640 10 monclient: _check_auth_tickets 2024-03-25T18:10:02.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.383+0000 7f617159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:02.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.383+0000 7f617159a640 1 -- 172.21.15.156:0/2258992805 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f6164001670 con 0x7f61740a98f0 2024-03-25T18:10:02.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.383+0000 7f6178e2e640 5 monclient: authenticate success, global_id 4297 2024-03-25T18:10:02.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.383+0000 7f6178e2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:02.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.383+0000 7f617159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:02.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.383+0000 7f617159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:02.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.383+0000 7f617159a640 10 monclient: dump: 2024-03-25T18:10:02.385 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:02.386 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:02.386 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:02.386 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:02.386 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:02.386 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:02.386 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:02.386 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:02.386 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:02.386 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:02.386 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.383+0000 7f6178e2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:02.393 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.392+0000 7f617159a640 1 -- 172.21.15.156:0/2258992805 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61640017d0 con 0x7f61740a98f0 2024-03-25T18:10:02.394 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.392+0000 7f617159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:02.394 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.392+0000 7f617159a640 1 -- 172.21.15.156:0/2258992805 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f6164001ba0 con 0x7f61740a98f0 2024-03-25T18:10:02.394 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.392+0000 7f617159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:02.394 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.392+0000 7f617159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:02.394 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.392+0000 7f617159a640 10 monclient: dump: 2024-03-25T18:10:02.394 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:02.394 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:02.394 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:02.394 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:02.394 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:02.394 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:02.394 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:02.394 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:02.395 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:02.395 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:02.395 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.392+0000 7f6178e2e640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:02.395 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.392+0000 7f6178e2e640 4 set_mon_vals no callback set 2024-03-25T18:10:02.396 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.393+0000 7f6178e2e640 10 monclient: shutdown 2024-03-25T18:10:02.396 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.393+0000 7f6178e2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:02.396 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.393+0000 7f6178e2e640 1 -- 172.21.15.156:0/2258992805 >> v1:172.21.15.194:6789/0 conn(0x7f61740a98f0 legacy=0x7f61740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:02.396 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.393+0000 7f6178e2e640 1 -- 172.21.15.156:0/2258992805 shutdown_connections 2024-03-25T18:10:02.396 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.393+0000 7f6178e2e640 1 -- 172.21.15.156:0/2258992805 >> 172.21.15.156:0/2258992805 conn(0x7f61740b7ea0 msgr2=0x7f61740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:02.396 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.393+0000 7f6178e2e640 1 -- 172.21.15.156:0/2258992805 shutdown_connections 2024-03-25T18:10:02.396 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.394+0000 7f6178e2e640 1 -- 172.21.15.156:0/2258992805 wait complete. 2024-03-25T18:10:02.396 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.394+0000 7f6178e2e640 10 monclient: build_initial_monmap 2024-03-25T18:10:02.396 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.394+0000 7f6178e2e640 10 monclient: monmap: 2024-03-25T18:10:02.396 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:10:02.397 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:02.397 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:10:02.397 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:10:02.397 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:02.397 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:10:02.397 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:10:02.397 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:10:02.397 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:10:02.397 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:02.397 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.395+0000 7f6178e2e640 1 librados: starting msgr at 2024-03-25T18:10:02.397 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.395+0000 7f6178e2e640 1 librados: starting objecter 2024-03-25T18:10:02.397 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.395+0000 7f6178e2e640 1 Processor -- start 2024-03-25T18:10:02.397 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.395+0000 7f6178e2e640 1 -- start start 2024-03-25T18:10:02.397 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.395+0000 7f6178e2e640 1 librados: setting wanted keys 2024-03-25T18:10:02.397 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.395+0000 7f6178e2e640 1 librados: calling monclient init 2024-03-25T18:10:02.397 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.395+0000 7f6178e2e640 10 monclient: init 2024-03-25T18:10:02.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.396+0000 7f6178e2e640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:02.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.396+0000 7f6178e2e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:10:02.399 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.396+0000 7f6178e2e640 10 monclient(hunting): picked mon.noname-b con 0x7f61740a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:10:02.399 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.396+0000 7f6178e2e640 10 monclient(hunting): picked mon.noname-a con 0x7f617413f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:10:02.399 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.396+0000 7f6178e2e640 10 monclient(hunting): picked mon.noname-c con 0x7f61741430c0 addr v1:172.21.15.156:6790/0 2024-03-25T18:10:02.399 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.396+0000 7f6178e2e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61741dabe0 con 0x7f617413f710 2024-03-25T18:10:02.399 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.396+0000 7f6178e2e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61741dbab0 con 0x7f61740a98f0 2024-03-25T18:10:02.399 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.396+0000 7f6178e2e640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61741d5060 con 0x7f61741430c0 2024-03-25T18:10:02.399 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.396+0000 7f6178e2e640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:02.399 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.396+0000 7f6178e2e640 10 monclient(hunting): authenticate will time out at 7083.706997s 2024-03-25T18:10:02.402 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.401+0000 7f617259c640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f61740a98f0 0x7f61741db3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:46878/0 (socket says 172.21.15.156:46878) 2024-03-25T18:10:02.402 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.401+0000 7f617259c640 1 -- 172.21.15.156:0/2129167559 learned_addr learned my addr 172.21.15.156:0/2129167559 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:10:02.403 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.402+0000 7f6162ffd640 10 client.?.objecter ms_handle_connect 0x7f61740a98f0 2024-03-25T18:10:02.403 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.402+0000 7f6162ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:02.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.408+0000 7f6162ffd640 10 client.?.objecter ms_handle_connect 0x7f61741430c0 2024-03-25T18:10:02.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.408+0000 7f6162ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:02.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.414+0000 7f6162ffd640 1 -- 172.21.15.156:0/2129167559 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4017946679 0 0) 0x7f61741dbab0 con 0x7f61740a98f0 2024-03-25T18:10:02.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.414+0000 7f6162ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:02.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.414+0000 7f6162ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:02.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.414+0000 7f6162ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:02.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.414+0000 7f6162ffd640 10 monclient(hunting): my global_id is 4300 2024-03-25T18:10:02.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.414+0000 7f6162ffd640 1 -- 172.21.15.156:0/2129167559 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6154004eb0 con 0x7f61740a98f0 2024-03-25T18:10:02.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.415+0000 7f6162ffd640 10 client.?.objecter ms_handle_connect 0x7f617413f710 2024-03-25T18:10:02.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.415+0000 7f6162ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:02.416 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.416+0000 7f6162ffd640 1 -- 172.21.15.156:0/2129167559 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1685132733 0 0) 0x7f61741d5060 con 0x7f61741430c0 2024-03-25T18:10:02.417 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.416+0000 7f6162ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:02.417 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.416+0000 7f6162ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:02.417 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.416+0000 7f6162ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:02.417 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.416+0000 7f6162ffd640 10 monclient(hunting): my global_id is 4310 2024-03-25T18:10:02.417 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.416+0000 7f6162ffd640 1 -- 172.21.15.156:0/2129167559 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61741dbab0 con 0x7f61741430c0 2024-03-25T18:10:02.418 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.418+0000 7f6162ffd640 1 -- 172.21.15.156:0/2129167559 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 973116981 0 0) 0x7f61741dabe0 con 0x7f617413f710 2024-03-25T18:10:02.419 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.418+0000 7f6162ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:02.419 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.418+0000 7f6162ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:02.419 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.418+0000 7f6162ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:02.419 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.418+0000 7f6162ffd640 10 monclient(hunting): my global_id is 4302 2024-03-25T18:10:02.419 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.418+0000 7f6162ffd640 1 -- 172.21.15.156:0/2129167559 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61741d5060 con 0x7f617413f710 2024-03-25T18:10:02.424 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.423+0000 7f6162ffd640 1 -- 172.21.15.156:0/2129167559 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3057736838 0 0) 0x7f61741d5060 con 0x7f617413f710 2024-03-25T18:10:02.424 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.423+0000 7f6162ffd640 1 -- 172.21.15.156:0/2129167559 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61741dabe0 con 0x7f617413f710 2024-03-25T18:10:02.424 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.423+0000 7f6162ffd640 1 -- 172.21.15.156:0/2129167559 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f6168001950 con 0x7f617413f710 2024-03-25T18:10:02.424 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.423+0000 7f6162ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:02.425 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.423+0000 7f6162ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:02.425 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.423+0000 7f6162ffd640 10 monclient(hunting): dump: 2024-03-25T18:10:02.425 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:02.425 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:02.425 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:02.425 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:02.425 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:02.425 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:02.425 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:02.425 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:02.425 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:02.425 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:02.425 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.423+0000 7f6162ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:02.429 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.428+0000 7f6162ffd640 1 -- 172.21.15.156:0/2129167559 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4081411076 0 0) 0x7f61741dbab0 con 0x7f61741430c0 2024-03-25T18:10:02.430 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.428+0000 7f6162ffd640 1 -- 172.21.15.156:0/2129167559 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61741d5060 con 0x7f61741430c0 2024-03-25T18:10:02.430 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.428+0000 7f6162ffd640 1 -- 172.21.15.156:0/2129167559 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f616c001b00 con 0x7f61741430c0 2024-03-25T18:10:02.430 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.428+0000 7f6162ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:02.430 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.428+0000 7f6162ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:02.430 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.428+0000 7f6162ffd640 10 monclient(hunting): dump: 2024-03-25T18:10:02.430 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:02.430 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:02.430 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:02.430 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:02.430 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:02.430 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:02.430 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:02.430 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:02.430 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:02.430 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:02.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.434+0000 7f6162ffd640 1 -- 172.21.15.156:0/2129167559 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2359825115 0 0) 0x7f61741dabe0 con 0x7f617413f710 2024-03-25T18:10:02.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.434+0000 7f6162ffd640 1 -- 172.21.15.156:0/2129167559 >> v1:172.21.15.156:6790/0 conn(0x7f61741430c0 legacy=0x7f6174138460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:02.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.434+0000 7f6162ffd640 1 -- 172.21.15.156:0/2129167559 >> v1:172.21.15.194:6789/0 conn(0x7f61740a98f0 legacy=0x7f61741db3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:02.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.434+0000 7f6162ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:10:02.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.434+0000 7f6162ffd640 1 monclient: found mon.a 2024-03-25T18:10:02.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.434+0000 7f6162ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:02.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.434+0000 7f6162ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:10:02.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.434+0000 7f6162ffd640 1 -- 172.21.15.156:0/2129167559 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61741d51f0 con 0x7f617413f710 2024-03-25T18:10:02.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.434+0000 7f6162ffd640 10 monclient: _finish_auth 0 2024-03-25T18:10:02.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.434+0000 7f6162ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:10:02.436 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.434+0000 7f6162ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:02.436 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.434+0000 7f6178e2e640 5 monclient: authenticate success, global_id 4302 2024-03-25T18:10:02.436 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.434+0000 7f6178e2e640 10 monclient: _renew_subs 2024-03-25T18:10:02.436 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.434+0000 7f6178e2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:10:02.436 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.434+0000 7f6178e2e640 1 -- 172.21.15.156:0/2129167559 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6174137fa0 con 0x7f617413f710 2024-03-25T18:10:02.436 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.434+0000 7f6178e2e640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:02.436 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.434+0000 7f6178e2e640 10 monclient: _renew_subs 2024-03-25T18:10:02.436 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.434+0000 7f6178e2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:10:02.436 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.434+0000 7f6178e2e640 1 -- 172.21.15.156:0/2129167559 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61741d43c0 con 0x7f617413f710 2024-03-25T18:10:02.436 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.434+0000 7f6178e2e640 1 librados: init done 2024-03-25T18:10:02.437 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.435+0000 7f6178e2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:02.437 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.435+0000 7f6178e2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:02.437 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.435+0000 7f6178e2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:10:02.437 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.435+0000 7f6178e2e640 1 -- 172.21.15.156:0/2129167559 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61741382a0 con 0x7f617413f710 2024-03-25T18:10:02.742 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.741+0000 7f6162ffd640 1 -- 172.21.15.156:0/2129167559 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6168003610 con 0x7f617413f710 2024-03-25T18:10:02.742 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.741+0000 7f6162ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:02.742 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.741+0000 7f6162ffd640 1 -- 172.21.15.156:0/2129167559 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f6168003780 con 0x7f617413f710 2024-03-25T18:10:02.742 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.741+0000 7f6162ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:02.742 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.741+0000 7f6162ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:02.742 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.741+0000 7f6162ffd640 10 monclient: dump: 2024-03-25T18:10:02.742 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:02.742 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:02.742 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:02.742 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:02.742 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:02.742 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:02.742 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:02.743 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:02.743 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:02.743 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:02.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.741+0000 7f6170d99640 4 set_mon_vals no callback set 2024-03-25T18:10:02.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.741+0000 7f6162ffd640 1 -- 172.21.15.156:0/2129167559 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f6168019750 con 0x7f617413f710 2024-03-25T18:10:02.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.742+0000 7f617259c640 10 monclient: get_auth_request con 0x7f615406df30 auth_method 0 2024-03-25T18:10:02.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.742+0000 7f6162ffd640 10 client.4302.objecter ms_handle_connect 0x7f615406df30 2024-03-25T18:10:02.753 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.752+0000 7f6162ffd640 1 -- 172.21.15.156:0/2129167559 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f6168053450 con 0x7f617413f710 2024-03-25T18:10:02.753 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.752+0000 7f6162ffd640 10 client.4302.objecter ms_dispatch 0x7f6174000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:02.753 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.752+0000 7f6162ffd640 3 client.4302.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:02.753 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.752+0000 7f6162ffd640 3 client.4302.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:02.753 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.752+0000 7f6162ffd640 20 client.4302.objecter dump_active .. 0 homeless 2024-03-25T18:10:02.753 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.752+0000 7f6162ffd640 1 -- 172.21.15.156:0/2129167559 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f616804ed90 con 0x7f617413f710 2024-03-25T18:10:02.753 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.752+0000 7f6162ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:02.753 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.752+0000 7f6162ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:02.997 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.996+0000 7f6178e2e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:02.997 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.996+0000 7f6178e2e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:02.997 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.996+0000 7f6178e2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:10:02.997 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:02.996+0000 7f6178e2e640 1 -- 172.21.15.156:0/2129167559 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61740627b0 con 0x7f617413f710 2024-03-25T18:10:03.018 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:03.017+0000 7f6162ffd640 1 -- 172.21.15.156:0/2129167559 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f6168050e90 con 0x7f617413f710 2024-03-25T18:10:03.018 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:03.017+0000 7f6162ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:03.018 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:03.017+0000 7f6162ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:03.018 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:10:03.018 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:03.021 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:03.020+0000 7f6178e2e640 10 librados: watch_flush enter 2024-03-25T18:10:03.021 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:03.020+0000 7f6178e2e640 10 librados: watch_flush exit 2024-03-25T18:10:03.021 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:03.020+0000 7f6178e2e640 20 client.4302.objecter shutdown clearing up homeless session... 2024-03-25T18:10:03.021 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:03.020+0000 7f6178e2e640 10 client.4302.objecter successfully canceled tick 2024-03-25T18:10:03.021 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:03.020+0000 7f6178e2e640 1 -- 172.21.15.156:0/2129167559 >> v1:172.21.15.194:6800/2946427713 conn(0x7f615406df30 legacy=0x7f61540703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:03.021 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:03.020+0000 7f6178e2e640 10 monclient: shutdown 2024-03-25T18:10:03.021 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:03.020+0000 7f6178e2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:03.022 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:03.020+0000 7f6178e2e640 1 -- 172.21.15.156:0/2129167559 >> v1:172.21.15.156:6789/0 conn(0x7f617413f710 legacy=0x7f61741d4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:03.022 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:03.020+0000 7f6178e2e640 1 -- 172.21.15.156:0/2129167559 shutdown_connections 2024-03-25T18:10:03.022 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:03.020+0000 7f6178e2e640 1 -- 172.21.15.156:0/2129167559 >> 172.21.15.156:0/2129167559 conn(0x7f61740b7ea0 msgr2=0x7f6174074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:03.022 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:03.020+0000 7f6178e2e640 1 -- 172.21.15.156:0/2129167559 shutdown_connections 2024-03-25T18:10:03.022 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:03.020+0000 7f6178e2e640 1 -- 172.21.15.156:0/2129167559 wait complete. 2024-03-25T18:10:03.022 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:03.020+0000 7f6178e2e640 1 librados: shutdown 2024-03-25T18:10:03.034 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:09.840 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:09.953 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.952+0000 7f747d97b640 10 monclient: get_monmap_and_config 2024-03-25T18:10:09.953 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.952+0000 7f747d97b640 10 monclient: build_initial_monmap 2024-03-25T18:10:09.953 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.952+0000 7f747d97b640 10 monclient: monmap: 2024-03-25T18:10:09.953 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:10:09.953 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:09.953 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:10:09.953441+0000 2024-03-25T18:10:09.953 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:10:09.953441+0000 2024-03-25T18:10:09.954 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:09.954 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:09.954 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:10:09.954 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:10:09.954 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:10:09.954 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:09.955 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.953+0000 7f747d97b640 1 Processor -- start 2024-03-25T18:10:09.955 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.953+0000 7f747d97b640 1 -- start start 2024-03-25T18:10:09.955 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.953+0000 7f747d97b640 10 monclient: init 2024-03-25T18:10:09.955 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.954+0000 7f747d97b640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:09.955 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.954+0000 7f747d97b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:10:09.955 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.954+0000 7f747d97b640 10 monclient(hunting): picked mon.noname-b con 0x7f74780a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:10:09.955 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.954+0000 7f747d97b640 10 monclient(hunting): picked mon.noname-a con 0x7f747813f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:10:09.955 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.954+0000 7f747d97b640 10 monclient(hunting): picked mon.noname-c con 0x7f74781430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:10:09.955 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.954+0000 7f747d97b640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7478142520 con 0x7f747813f710 2024-03-25T18:10:09.955 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.954+0000 7f747d97b640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74780b28a0 con 0x7f74781430c0 2024-03-25T18:10:09.955 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.954+0000 7f747d97b640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74780b2a30 con 0x7f74780a98f0 2024-03-25T18:10:09.955 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.954+0000 7f747d97b640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:09.955 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.954+0000 7f747d97b640 10 monclient(hunting): authenticate will time out at 7091.265254s 2024-03-25T18:10:09.966 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.965+0000 7f74777fe640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f74781430c0 0x7f747814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:51746/0 (socket says 172.21.15.156:51746) 2024-03-25T18:10:09.966 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.965+0000 7f74777fe640 1 -- 172.21.15.156:0/1914369889 learned_addr learned my addr 172.21.15.156:0/1914369889 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:10:09.979 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.978+0000 7f7475ffb640 1 -- 172.21.15.156:0/1914369889 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2035854425 0 0) 0x7f74780b28a0 con 0x7f74781430c0 2024-03-25T18:10:09.979 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.978+0000 7f7475ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:09.979 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.978+0000 7f7475ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:09.979 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.978+0000 7f7475ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:09.979 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.978+0000 7f7475ffb640 10 monclient(hunting): my global_id is 4303 2024-03-25T18:10:09.979 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.978+0000 7f7475ffb640 1 -- 172.21.15.156:0/1914369889 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7460003520 con 0x7f74781430c0 2024-03-25T18:10:09.982 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.981+0000 7f7475ffb640 1 -- 172.21.15.156:0/1914369889 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 184911705 0 0) 0x7f7478142520 con 0x7f747813f710 2024-03-25T18:10:09.982 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.981+0000 7f7475ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:09.982 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.981+0000 7f7475ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:09.982 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.981+0000 7f7475ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:09.982 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.981+0000 7f7475ffb640 10 monclient(hunting): my global_id is 4305 2024-03-25T18:10:09.982 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.981+0000 7f7475ffb640 1 -- 172.21.15.156:0/1914369889 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74780b28a0 con 0x7f747813f710 2024-03-25T18:10:09.986 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.985+0000 7f7475ffb640 1 -- 172.21.15.156:0/1914369889 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242026612 0 0) 0x7f74780b2a30 con 0x7f74780a98f0 2024-03-25T18:10:09.986 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.985+0000 7f7475ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:09.986 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.985+0000 7f7475ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:09.986 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.985+0000 7f7475ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:09.986 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.985+0000 7f7475ffb640 10 monclient(hunting): my global_id is 4313 2024-03-25T18:10:09.987 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.985+0000 7f7475ffb640 1 -- 172.21.15.156:0/1914369889 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7478142520 con 0x7f74780a98f0 2024-03-25T18:10:09.990 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.989+0000 7f7475ffb640 1 -- 172.21.15.156:0/1914369889 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 91033166 0 0) 0x7f7460003520 con 0x7f74781430c0 2024-03-25T18:10:09.990 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.989+0000 7f7475ffb640 1 -- 172.21.15.156:0/1914369889 >> v1:172.21.15.156:6790/0 conn(0x7f74780a98f0 legacy=0x7f74780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:09.991 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.989+0000 7f7475ffb640 1 -- 172.21.15.156:0/1914369889 >> v1:172.21.15.156:6789/0 conn(0x7f747813f710 legacy=0x7f74781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:09.991 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.989+0000 7f7475ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:10:09.991 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.989+0000 7f7475ffb640 1 monclient: found mon.noname-c 2024-03-25T18:10:09.991 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.989+0000 7f7475ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:09.991 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.989+0000 7f7475ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:10:09.991 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.989+0000 7f7475ffb640 1 -- 172.21.15.156:0/1914369889 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74780b44c0 con 0x7f74781430c0 2024-03-25T18:10:09.991 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.989+0000 7f7475ffb640 10 monclient: _finish_auth 0 2024-03-25T18:10:09.991 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.989+0000 7f7475ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:10:09.991 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.989+0000 7f7475ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:09.991 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.989+0000 7f7475ffb640 1 -- 172.21.15.156:0/1914369889 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f746c001690 con 0x7f74781430c0 2024-03-25T18:10:09.991 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.989+0000 7f7475ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:09.991 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.989+0000 7f7475ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:09.991 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.989+0000 7f7475ffb640 10 monclient: dump: 2024-03-25T18:10:09.991 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:09.991 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:09.992 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:09.992 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:09.992 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:09.992 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:09.992 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:09.992 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:09.992 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:09.992 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:09.992 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.989+0000 7f747d97b640 5 monclient: authenticate success, global_id 4303 2024-03-25T18:10:09.992 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.989+0000 7f747d97b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:09.993 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.992+0000 7f7475ffb640 1 -- 172.21.15.156:0/1914369889 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f746c0017f0 con 0x7f74781430c0 2024-03-25T18:10:09.993 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.992+0000 7f7475ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:09.993 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.992+0000 7f747d97b640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:09.993 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.992+0000 7f747d97b640 4 set_mon_vals no callback set 2024-03-25T18:10:09.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.993+0000 7f747d97b640 10 monclient: shutdown 2024-03-25T18:10:09.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.993+0000 7f747d97b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:09.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.993+0000 7f747d97b640 1 -- 172.21.15.156:0/1914369889 >> v1:172.21.15.194:6789/0 conn(0x7f74781430c0 legacy=0x7f747814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:09.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.993+0000 7f747d97b640 1 -- 172.21.15.156:0/1914369889 shutdown_connections 2024-03-25T18:10:09.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.993+0000 7f747d97b640 1 -- 172.21.15.156:0/1914369889 >> 172.21.15.156:0/1914369889 conn(0x7f74780b7ea0 msgr2=0x7f74780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:09.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.993+0000 7f747d97b640 1 -- 172.21.15.156:0/1914369889 shutdown_connections 2024-03-25T18:10:09.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.993+0000 7f747d97b640 1 -- 172.21.15.156:0/1914369889 wait complete. 2024-03-25T18:10:09.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.994+0000 7f747d97b640 10 monclient: build_initial_monmap 2024-03-25T18:10:09.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.994+0000 7f747d97b640 10 monclient: monmap: 2024-03-25T18:10:09.995 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:10:09.995 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:09.995 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:10:09.995 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:10:09.995 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:09.996 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:10:09.996 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:10:09.996 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:10:09.996 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:10:09.996 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:09.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.994+0000 7f747d97b640 1 librados: starting msgr at 2024-03-25T18:10:09.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.994+0000 7f747d97b640 1 librados: starting objecter 2024-03-25T18:10:09.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.994+0000 7f747d97b640 1 Processor -- start 2024-03-25T18:10:09.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.995+0000 7f747d97b640 1 -- start start 2024-03-25T18:10:09.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.995+0000 7f747d97b640 1 librados: setting wanted keys 2024-03-25T18:10:09.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.995+0000 7f747d97b640 1 librados: calling monclient init 2024-03-25T18:10:09.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.995+0000 7f747d97b640 10 monclient: init 2024-03-25T18:10:09.997 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.995+0000 7f747d97b640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:09.997 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.995+0000 7f747d97b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:10:09.997 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.995+0000 7f747d97b640 10 monclient(hunting): picked mon.noname-b con 0x7f74780a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:10:09.997 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.995+0000 7f747d97b640 10 monclient(hunting): picked mon.noname-c con 0x7f747813f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:10:09.998 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.995+0000 7f747d97b640 10 monclient(hunting): picked mon.noname-a con 0x7f74781430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:10:09.998 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.995+0000 7f747d97b640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74780ad6e0 con 0x7f74781430c0 2024-03-25T18:10:09.998 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.995+0000 7f747d97b640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74780b0ab0 con 0x7f74780a98f0 2024-03-25T18:10:09.998 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.995+0000 7f747d97b640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74780afe70 con 0x7f747813f710 2024-03-25T18:10:09.998 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.995+0000 7f747d97b640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:09.998 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:09.995+0000 7f747d97b640 10 monclient(hunting): authenticate will time out at 7091.306750s 2024-03-25T18:10:10.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.006+0000 7f7476ffd640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f74780a98f0 0x7f74780abfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:51764/0 (socket says 172.21.15.156:51764) 2024-03-25T18:10:10.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.006+0000 7f7476ffd640 1 -- 172.21.15.156:0/1002137105 learned_addr learned my addr 172.21.15.156:0/1002137105 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:10:10.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.008+0000 7f745ffff640 10 client.?.objecter ms_handle_connect 0x7f74780a98f0 2024-03-25T18:10:10.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.008+0000 7f745ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:10.017 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.016+0000 7f745ffff640 10 client.?.objecter ms_handle_connect 0x7f747813f710 2024-03-25T18:10:10.017 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.016+0000 7f745ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:10.017 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.016+0000 7f745ffff640 10 client.?.objecter ms_handle_connect 0x7f74781430c0 2024-03-25T18:10:10.018 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.016+0000 7f745ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:10.019 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.018+0000 7f745ffff640 1 -- 172.21.15.156:0/1002137105 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429794620 0 0) 0x7f74780b0ab0 con 0x7f74780a98f0 2024-03-25T18:10:10.019 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.018+0000 7f745ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:10.019 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.018+0000 7f745ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:10.019 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.018+0000 7f745ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:10.019 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.018+0000 7f745ffff640 10 monclient(hunting): my global_id is 4306 2024-03-25T18:10:10.019 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.018+0000 7f745ffff640 1 -- 172.21.15.156:0/1002137105 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7464004eb0 con 0x7f74780a98f0 2024-03-25T18:10:10.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.035+0000 7f745ffff640 1 -- 172.21.15.156:0/1002137105 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2361233202 0 0) 0x7f7464004eb0 con 0x7f74780a98f0 2024-03-25T18:10:10.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.035+0000 7f745ffff640 1 -- 172.21.15.156:0/1002137105 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74780b0ab0 con 0x7f74780a98f0 2024-03-25T18:10:10.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.035+0000 7f745ffff640 1 -- 172.21.15.156:0/1002137105 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7468001a10 con 0x7f74780a98f0 2024-03-25T18:10:10.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.035+0000 7f745ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:10.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.035+0000 7f745ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:10.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.035+0000 7f745ffff640 10 monclient(hunting): dump: 2024-03-25T18:10:10.037 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:10.037 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:10.037 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:10.037 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:10.037 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:10.037 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:10.037 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:10.037 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:10.037 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:10.037 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:10.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.035+0000 7f745ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:10.040 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.039+0000 7f745ffff640 1 -- 172.21.15.156:0/1002137105 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350745001 0 0) 0x7f74780ad6e0 con 0x7f74781430c0 2024-03-25T18:10:10.040 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.039+0000 7f745ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:10.040 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.039+0000 7f745ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:10.040 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.039+0000 7f745ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:10.040 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.039+0000 7f745ffff640 10 monclient(hunting): my global_id is 4308 2024-03-25T18:10:10.040 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.039+0000 7f745ffff640 1 -- 172.21.15.156:0/1002137105 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7464004eb0 con 0x7f74781430c0 2024-03-25T18:10:10.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.041+0000 7f745ffff640 1 -- 172.21.15.156:0/1002137105 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 531939791 0 0) 0x7f74780b0ab0 con 0x7f74780a98f0 2024-03-25T18:10:10.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.041+0000 7f745ffff640 1 -- 172.21.15.156:0/1002137105 >> v1:172.21.15.156:6790/0 conn(0x7f747813f710 legacy=0x7f74781df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:10.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.041+0000 7f745ffff640 1 -- 172.21.15.156:0/1002137105 >> v1:172.21.15.156:6789/0 conn(0x7f74781430c0 legacy=0x7f74781e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:10.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.041+0000 7f745ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:10:10.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.041+0000 7f745ffff640 1 monclient: found mon.b 2024-03-25T18:10:10.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.041+0000 7f745ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:10.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.041+0000 7f745ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:10:10.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.041+0000 7f745ffff640 1 -- 172.21.15.156:0/1002137105 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74780b0d20 con 0x7f74780a98f0 2024-03-25T18:10:10.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.041+0000 7f745ffff640 10 monclient: _finish_auth 0 2024-03-25T18:10:10.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.041+0000 7f745ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:10:10.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.041+0000 7f745ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:10.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.041+0000 7f747d97b640 5 monclient: authenticate success, global_id 4306 2024-03-25T18:10:10.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.041+0000 7f747d97b640 10 monclient: _renew_subs 2024-03-25T18:10:10.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.041+0000 7f747d97b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:10:10.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.041+0000 7f747d97b640 1 -- 172.21.15.156:0/1002137105 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74780b1000 con 0x7f74780a98f0 2024-03-25T18:10:10.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.041+0000 7f747d97b640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:10.044 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.041+0000 7f747d97b640 10 monclient: _renew_subs 2024-03-25T18:10:10.044 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.041+0000 7f747d97b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:10:10.044 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.041+0000 7f747d97b640 1 -- 172.21.15.156:0/1002137105 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74781434a0 con 0x7f74780a98f0 2024-03-25T18:10:10.044 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.041+0000 7f747d97b640 1 librados: init done 2024-03-25T18:10:10.044 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.042+0000 7f747d97b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:10.044 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.042+0000 7f747d97b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:10.045 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.042+0000 7f747d97b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:10:10.045 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.042+0000 7f747d97b640 1 -- 172.21.15.156:0/1002137105 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74780afaa0 con 0x7f74780a98f0 2024-03-25T18:10:10.321 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.320+0000 7f745ffff640 1 -- 172.21.15.156:0/1002137105 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7468001cb0 con 0x7f74780a98f0 2024-03-25T18:10:10.321 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.320+0000 7f745ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:10.321 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.320+0000 7f745ffff640 1 -- 172.21.15.156:0/1002137105 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7468003a60 con 0x7f74780a98f0 2024-03-25T18:10:10.321 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.320+0000 7f745ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:10.321 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.320+0000 7f745ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:10.321 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.320+0000 7f745ffff640 10 monclient: dump: 2024-03-25T18:10:10.321 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:10.321 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:10.321 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:10.322 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:10.322 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:10.322 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:10.322 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:10.322 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:10.322 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:10.322 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:10.323 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.320+0000 7f74757fa640 4 set_mon_vals no callback set 2024-03-25T18:10:10.323 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.321+0000 7f745ffff640 1 -- 172.21.15.156:0/1002137105 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f7468003c80 con 0x7f74780a98f0 2024-03-25T18:10:10.323 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.322+0000 7f74767fc640 10 monclient: get_auth_request con 0x7f746406dd90 auth_method 0 2024-03-25T18:10:10.323 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.322+0000 7f745ffff640 10 client.4306.objecter ms_handle_connect 0x7f746406dd90 2024-03-25T18:10:10.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.325+0000 7f745ffff640 1 -- 172.21.15.156:0/1002137105 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f7468054850 con 0x7f74780a98f0 2024-03-25T18:10:10.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.325+0000 7f745ffff640 10 client.4306.objecter ms_dispatch 0x7f7478000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:10.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.325+0000 7f745ffff640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:10.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.325+0000 7f745ffff640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:10.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.326+0000 7f745ffff640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-25T18:10:10.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.326+0000 7f745ffff640 1 -- 172.21.15.156:0/1002137105 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7468050360 con 0x7f74780a98f0 2024-03-25T18:10:10.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.326+0000 7f745ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:10.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.326+0000 7f745ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:10.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.553+0000 7f747d97b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:10.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.553+0000 7f747d97b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:10.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.553+0000 7f747d97b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:10:10.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.553+0000 7f747d97b640 1 -- 172.21.15.156:0/1002137105 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74780627b0 con 0x7f74780a98f0 2024-03-25T18:10:10.687 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.686+0000 7f745ffff640 1 -- 172.21.15.156:0/1002137105 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f7468052290 con 0x7f74780a98f0 2024-03-25T18:10:10.687 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.686+0000 7f745ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:10.688 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.686+0000 7f745ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:10.688 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:10:10.688 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:10.690 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.689+0000 7f747d97b640 10 librados: watch_flush enter 2024-03-25T18:10:10.690 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.689+0000 7f747d97b640 10 librados: watch_flush exit 2024-03-25T18:10:10.690 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.689+0000 7f747d97b640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-25T18:10:10.691 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.689+0000 7f747d97b640 10 client.4306.objecter successfully canceled tick 2024-03-25T18:10:10.691 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.689+0000 7f747d97b640 1 -- 172.21.15.156:0/1002137105 >> v1:172.21.15.194:6800/2946427713 conn(0x7f746406dd90 legacy=0x7f7464070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:10.691 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.689+0000 7f747d97b640 10 monclient: shutdown 2024-03-25T18:10:10.691 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.689+0000 7f747d97b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:10.691 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.689+0000 7f747d97b640 1 -- 172.21.15.156:0/1002137105 >> v1:172.21.15.194:6789/0 conn(0x7f74780a98f0 legacy=0x7f74780abfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:10.691 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.689+0000 7f747d97b640 1 -- 172.21.15.156:0/1002137105 shutdown_connections 2024-03-25T18:10:10.691 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.689+0000 7f747d97b640 1 -- 172.21.15.156:0/1002137105 >> 172.21.15.156:0/1002137105 conn(0x7f74780b7ea0 msgr2=0x7f747813c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:10.691 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.689+0000 7f747d97b640 1 -- 172.21.15.156:0/1002137105 shutdown_connections 2024-03-25T18:10:10.691 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.689+0000 7f747d97b640 1 -- 172.21.15.156:0/1002137105 wait complete. 2024-03-25T18:10:10.691 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:10.689+0000 7f747d97b640 1 librados: shutdown 2024-03-25T18:10:10.702 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:13.156 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:02:19.146 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:13.207 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:02:19.198 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:13.236 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:02:19.227 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:13.247 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:02:19.214 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:13.251 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:02:19.242 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:13.262 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:02:19.253 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:13.263 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:02:19.231 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:13.293 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:02:19.260 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:13.299 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:02:19.290 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:17.509 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:17.627 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.626+0000 7f15c7b7c640 10 monclient: get_monmap_and_config 2024-03-25T18:10:17.627 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.626+0000 7f15c7b7c640 10 monclient: build_initial_monmap 2024-03-25T18:10:17.627 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.626+0000 7f15c7b7c640 10 monclient: monmap: 2024-03-25T18:10:17.627 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:10:17.627 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:17.627 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:10:17.627214+0000 2024-03-25T18:10:17.628 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:10:17.627214+0000 2024-03-25T18:10:17.628 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:17.628 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:17.628 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:10:17.628 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:10:17.628 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:10:17.628 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:17.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.627+0000 7f15c7b7c640 1 Processor -- start 2024-03-25T18:10:17.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.627+0000 7f15c7b7c640 1 -- start start 2024-03-25T18:10:17.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.627+0000 7f15c7b7c640 10 monclient: init 2024-03-25T18:10:17.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.627+0000 7f15c7b7c640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:17.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.627+0000 7f15c7b7c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:10:17.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.627+0000 7f15c7b7c640 10 monclient(hunting): picked mon.noname-a con 0x7f15c00a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:10:17.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.627+0000 7f15c7b7c640 10 monclient(hunting): picked mon.noname-c con 0x7f15c013f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:10:17.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.627+0000 7f15c7b7c640 10 monclient(hunting): picked mon.noname-b con 0x7f15c01430c0 addr v1:172.21.15.156:6790/0 2024-03-25T18:10:17.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.628+0000 7f15c7b7c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c0142520 con 0x7f15c00a98f0 2024-03-25T18:10:17.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.628+0000 7f15c7b7c640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c00b28a0 con 0x7f15c013f710 2024-03-25T18:10:17.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.628+0000 7f15c7b7c640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c00b2a30 con 0x7f15c01430c0 2024-03-25T18:10:17.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.628+0000 7f15c7b7c640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:17.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.628+0000 7f15c7b7c640 10 monclient(hunting): authenticate will time out at 7098.938831s 2024-03-25T18:10:17.639 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.638+0000 7f15c5918640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f15c00a98f0 0x7f15c00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44902/0 (socket says 172.21.15.156:44902) 2024-03-25T18:10:17.639 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.638+0000 7f15c5918640 1 -- 172.21.15.156:0/4033849997 learned_addr learned my addr 172.21.15.156:0/4033849997 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:10:17.649 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.648+0000 7f15c4916640 1 -- 172.21.15.156:0/4033849997 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736095922 0 0) 0x7f15c0142520 con 0x7f15c00a98f0 2024-03-25T18:10:17.649 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.648+0000 7f15c4916640 10 monclient(hunting): _negotiate 2024-03-25T18:10:17.649 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.648+0000 7f15c4916640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:17.649 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.648+0000 7f15c4916640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:17.650 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.648+0000 7f15c4916640 10 monclient(hunting): my global_id is 4311 2024-03-25T18:10:17.650 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.648+0000 7f15c4916640 1 -- 172.21.15.156:0/4033849997 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15ac003520 con 0x7f15c00a98f0 2024-03-25T18:10:17.650 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.648+0000 7f15c4916640 1 -- 172.21.15.156:0/4033849997 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3896332561 0 0) 0x7f15c00b28a0 con 0x7f15c013f710 2024-03-25T18:10:17.650 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.648+0000 7f15c4916640 10 monclient(hunting): _negotiate 2024-03-25T18:10:17.650 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.648+0000 7f15c4916640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:17.650 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.648+0000 7f15c4916640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:17.650 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.648+0000 7f15c4916640 10 monclient(hunting): my global_id is 4309 2024-03-25T18:10:17.650 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.648+0000 7f15c4916640 1 -- 172.21.15.156:0/4033849997 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15c0142520 con 0x7f15c013f710 2024-03-25T18:10:17.656 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.655+0000 7f15c4916640 1 -- 172.21.15.156:0/4033849997 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1022579225 0 0) 0x7f15c0142520 con 0x7f15c013f710 2024-03-25T18:10:17.657 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.655+0000 7f15c4916640 1 -- 172.21.15.156:0/4033849997 >> v1:172.21.15.156:6790/0 conn(0x7f15c01430c0 legacy=0x7f15c014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:17.657 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.655+0000 7f15c4916640 1 -- 172.21.15.156:0/4033849997 >> v1:172.21.15.156:6789/0 conn(0x7f15c00a98f0 legacy=0x7f15c00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:17.657 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.655+0000 7f15c4916640 10 monclient: _finish_hunting 0 2024-03-25T18:10:17.657 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.655+0000 7f15c4916640 1 monclient: found mon.noname-c 2024-03-25T18:10:17.657 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.655+0000 7f15c4916640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:17.657 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.655+0000 7f15c4916640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:10:17.657 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.655+0000 7f15c4916640 1 -- 172.21.15.156:0/4033849997 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c00b44c0 con 0x7f15c013f710 2024-03-25T18:10:17.657 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.655+0000 7f15c4916640 10 monclient: _finish_auth 0 2024-03-25T18:10:17.657 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.655+0000 7f15c4916640 10 monclient: _check_auth_tickets 2024-03-25T18:10:17.657 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.655+0000 7f15c4916640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:17.658 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.655+0000 7f15c4916640 1 -- 172.21.15.156:0/4033849997 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f15b0001670 con 0x7f15c013f710 2024-03-25T18:10:17.658 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.655+0000 7f15c4916640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:17.658 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.655+0000 7f15c4916640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:17.658 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.655+0000 7f15c4916640 10 monclient: dump: 2024-03-25T18:10:17.658 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:17.658 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:17.658 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:17.658 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:17.658 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:17.658 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:17.658 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:17.659 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:17.659 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:17.659 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:17.659 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.655+0000 7f15c7b7c640 5 monclient: authenticate success, global_id 4309 2024-03-25T18:10:17.659 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.655+0000 7f15c7b7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:17.696 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.695+0000 7f15c4916640 1 -- 172.21.15.156:0/4033849997 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15b00017d0 con 0x7f15c013f710 2024-03-25T18:10:17.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.695+0000 7f15c4916640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:17.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.695+0000 7f15c4916640 1 -- 172.21.15.156:0/4033849997 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f15b0001ba0 con 0x7f15c013f710 2024-03-25T18:10:17.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.695+0000 7f15c4916640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:17.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.696+0000 7f15c4916640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:17.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.696+0000 7f15c4916640 10 monclient: dump: 2024-03-25T18:10:17.697 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:17.697 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:17.697 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:17.697 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:17.697 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:17.697 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:17.698 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:17.698 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:17.698 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:17.698 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:17.698 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.696+0000 7f15c7b7c640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:17.698 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.696+0000 7f15c7b7c640 4 set_mon_vals no callback set 2024-03-25T18:10:17.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.696+0000 7f15c7b7c640 10 monclient: shutdown 2024-03-25T18:10:17.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.696+0000 7f15c7b7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:17.701 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.696+0000 7f15c7b7c640 1 -- 172.21.15.156:0/4033849997 >> v1:172.21.15.194:6789/0 conn(0x7f15c013f710 legacy=0x7f15c01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:17.701 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.696+0000 7f15c7b7c640 1 -- 172.21.15.156:0/4033849997 shutdown_connections 2024-03-25T18:10:17.701 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.697+0000 7f15c7b7c640 1 -- 172.21.15.156:0/4033849997 >> 172.21.15.156:0/4033849997 conn(0x7f15c00b7ea0 msgr2=0x7f15c00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:17.701 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.697+0000 7f15c7b7c640 1 -- 172.21.15.156:0/4033849997 shutdown_connections 2024-03-25T18:10:17.701 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.697+0000 7f15c7b7c640 1 -- 172.21.15.156:0/4033849997 wait complete. 2024-03-25T18:10:17.701 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.697+0000 7f15c7b7c640 10 monclient: build_initial_monmap 2024-03-25T18:10:17.701 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.697+0000 7f15c7b7c640 10 monclient: monmap: 2024-03-25T18:10:17.701 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:10:17.701 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:17.701 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:10:17.701 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:10:17.701 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:17.702 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:10:17.702 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:10:17.702 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:10:17.702 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:10:17.702 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:17.702 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.698+0000 7f15c7b7c640 1 librados: starting msgr at 2024-03-25T18:10:17.702 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.698+0000 7f15c7b7c640 1 librados: starting objecter 2024-03-25T18:10:17.702 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.698+0000 7f15c7b7c640 1 Processor -- start 2024-03-25T18:10:17.702 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.698+0000 7f15c7b7c640 1 -- start start 2024-03-25T18:10:17.702 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.698+0000 7f15c7b7c640 1 librados: setting wanted keys 2024-03-25T18:10:17.702 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.698+0000 7f15c7b7c640 1 librados: calling monclient init 2024-03-25T18:10:17.702 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.698+0000 7f15c7b7c640 10 monclient: init 2024-03-25T18:10:17.703 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.699+0000 7f15c7b7c640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:17.703 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.699+0000 7f15c7b7c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:10:17.703 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.699+0000 7f15c7b7c640 10 monclient(hunting): picked mon.noname-a con 0x7f15c00a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:10:17.703 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.699+0000 7f15c7b7c640 10 monclient(hunting): picked mon.noname-b con 0x7f15c013f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:10:17.703 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.699+0000 7f15c7b7c640 10 monclient(hunting): picked mon.noname-c con 0x7f15c01430c0 addr v1:172.21.15.156:6790/0 2024-03-25T18:10:17.703 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.699+0000 7f15c7b7c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c00abae0 con 0x7f15c00a98f0 2024-03-25T18:10:17.703 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.699+0000 7f15c7b7c640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c00acb60 con 0x7f15c013f710 2024-03-25T18:10:17.703 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.699+0000 7f15c7b7c640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c00abd50 con 0x7f15c01430c0 2024-03-25T18:10:17.703 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.699+0000 7f15c7b7c640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:17.703 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.699+0000 7f15c7b7c640 10 monclient(hunting): authenticate will time out at 7099.009889s 2024-03-25T18:10:17.714 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.713+0000 7f15c5117640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f15c013f710 0x7f15c00ac450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:59218/0 (socket says 172.21.15.156:59218) 2024-03-25T18:10:17.714 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.713+0000 7f15c5117640 1 -- 172.21.15.156:0/3012496183 learned_addr learned my addr 172.21.15.156:0/3012496183 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:10:17.715 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.715+0000 7f15b67fc640 10 client.?.objecter ms_handle_connect 0x7f15c013f710 2024-03-25T18:10:17.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.715+0000 7f15b67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:17.722 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.721+0000 7f15b67fc640 10 client.?.objecter ms_handle_connect 0x7f15c00a98f0 2024-03-25T18:10:17.722 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.721+0000 7f15b67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:17.722 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.722+0000 7f15b67fc640 10 client.?.objecter ms_handle_connect 0x7f15c01430c0 2024-03-25T18:10:17.723 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.722+0000 7f15b67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:17.727 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.726+0000 7f15b67fc640 1 -- 172.21.15.156:0/3012496183 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 957505366 0 0) 0x7f15c00abd50 con 0x7f15c01430c0 2024-03-25T18:10:17.727 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.727+0000 7f15b67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:10:17.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.727+0000 7f15b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:17.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.727+0000 7f15b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:17.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.727+0000 7f15b67fc640 10 monclient(hunting): my global_id is 4322 2024-03-25T18:10:17.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.727+0000 7f15b67fc640 1 -- 172.21.15.156:0/3012496183 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15a4004eb0 con 0x7f15c01430c0 2024-03-25T18:10:17.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.729+0000 7f15b67fc640 1 -- 172.21.15.156:0/3012496183 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925491233 0 0) 0x7f15c00abae0 con 0x7f15c00a98f0 2024-03-25T18:10:17.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.729+0000 7f15b67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:10:17.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.729+0000 7f15b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:17.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.729+0000 7f15b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:17.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.729+0000 7f15b67fc640 10 monclient(hunting): my global_id is 4314 2024-03-25T18:10:17.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.729+0000 7f15b67fc640 1 -- 172.21.15.156:0/3012496183 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15c00abd50 con 0x7f15c00a98f0 2024-03-25T18:10:17.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.731+0000 7f15b67fc640 1 -- 172.21.15.156:0/3012496183 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3000574940 0 0) 0x7f15c00acb60 con 0x7f15c013f710 2024-03-25T18:10:17.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.731+0000 7f15b67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:10:17.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.731+0000 7f15b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:17.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.731+0000 7f15b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:17.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.731+0000 7f15b67fc640 10 monclient(hunting): my global_id is 4312 2024-03-25T18:10:17.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.731+0000 7f15b67fc640 1 -- 172.21.15.156:0/3012496183 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15c00abae0 con 0x7f15c013f710 2024-03-25T18:10:17.742 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.740+0000 7f15b67fc640 1 -- 172.21.15.156:0/3012496183 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1936164662 0 0) 0x7f15c00abd50 con 0x7f15c00a98f0 2024-03-25T18:10:17.742 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.740+0000 7f15b67fc640 1 -- 172.21.15.156:0/3012496183 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15c00acb60 con 0x7f15c00a98f0 2024-03-25T18:10:17.742 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.740+0000 7f15b67fc640 1 -- 172.21.15.156:0/3012496183 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f15bc0018d0 con 0x7f15c00a98f0 2024-03-25T18:10:17.742 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.740+0000 7f15b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:17.742 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.740+0000 7f15b67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:17.742 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.740+0000 7f15b67fc640 10 monclient(hunting): dump: 2024-03-25T18:10:17.742 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:17.742 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:17.742 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:17.742 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:17.742 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:17.743 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:17.743 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:17.743 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:17.743 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:17.743 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:17.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.741+0000 7f15b67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:17.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.744+0000 7f15b67fc640 1 -- 172.21.15.156:0/3012496183 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 419362013 0 0) 0x7f15a4004eb0 con 0x7f15c01430c0 2024-03-25T18:10:17.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.744+0000 7f15b67fc640 1 -- 172.21.15.156:0/3012496183 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15c00abd50 con 0x7f15c01430c0 2024-03-25T18:10:17.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.744+0000 7f15b67fc640 1 -- 172.21.15.156:0/3012496183 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f15b8001b20 con 0x7f15c01430c0 2024-03-25T18:10:17.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.744+0000 7f15b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:17.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.744+0000 7f15b67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:17.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.744+0000 7f15b67fc640 10 monclient(hunting): dump: 2024-03-25T18:10:17.745 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:17.746 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:17.746 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:17.746 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:17.746 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:17.746 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:17.746 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:17.746 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:17.746 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:17.746 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:17.747 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.746+0000 7f15b67fc640 1 -- 172.21.15.156:0/3012496183 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1591852941 0 0) 0x7f15c00abae0 con 0x7f15c013f710 2024-03-25T18:10:17.747 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.746+0000 7f15b67fc640 1 -- 172.21.15.156:0/3012496183 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15a4004eb0 con 0x7f15c013f710 2024-03-25T18:10:17.747 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.746+0000 7f15b67fc640 1 -- 172.21.15.156:0/3012496183 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f15b0001ba0 con 0x7f15c013f710 2024-03-25T18:10:17.747 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.746+0000 7f15b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:17.747 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.746+0000 7f15b67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:17.747 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.746+0000 7f15b67fc640 10 monclient(hunting): dump: 2024-03-25T18:10:17.747 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:17.747 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:17.747 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:17.748 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:17.748 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:17.748 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:17.748 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:17.748 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:17.748 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:17.748 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:17.750 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.749+0000 7f15b67fc640 1 -- 172.21.15.156:0/3012496183 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1056512711 0 0) 0x7f15c00acb60 con 0x7f15c00a98f0 2024-03-25T18:10:17.750 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.749+0000 7f15b67fc640 1 -- 172.21.15.156:0/3012496183 >> v1:172.21.15.156:6790/0 conn(0x7f15c01430c0 legacy=0x7f15c01e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:17.750 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.749+0000 7f15b67fc640 1 -- 172.21.15.156:0/3012496183 >> v1:172.21.15.194:6789/0 conn(0x7f15c013f710 legacy=0x7f15c00ac450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:17.750 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.749+0000 7f15b67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:10:17.750 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.749+0000 7f15b67fc640 1 monclient: found mon.a 2024-03-25T18:10:17.750 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.749+0000 7f15b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:17.751 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.749+0000 7f15b67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:10:17.751 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.749+0000 7f15b67fc640 1 -- 172.21.15.156:0/3012496183 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c00ac030 con 0x7f15c00a98f0 2024-03-25T18:10:17.751 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.749+0000 7f15b67fc640 10 monclient: _finish_auth 0 2024-03-25T18:10:17.751 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.749+0000 7f15b67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:10:17.751 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.749+0000 7f15b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:17.751 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.749+0000 7f15c7b7c640 5 monclient: authenticate success, global_id 4314 2024-03-25T18:10:17.751 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.749+0000 7f15c7b7c640 10 monclient: _renew_subs 2024-03-25T18:10:17.751 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.749+0000 7f15c7b7c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:10:17.751 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.749+0000 7f15c7b7c640 1 -- 172.21.15.156:0/3012496183 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15c01390d0 con 0x7f15c00a98f0 2024-03-25T18:10:17.751 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.749+0000 7f15c7b7c640 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:17.751 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.749+0000 7f15c7b7c640 10 monclient: _renew_subs 2024-03-25T18:10:17.751 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.749+0000 7f15c7b7c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:10:17.751 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.749+0000 7f15c7b7c640 1 -- 172.21.15.156:0/3012496183 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c0136520 con 0x7f15c00a98f0 2024-03-25T18:10:17.751 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.749+0000 7f15c7b7c640 1 librados: init done 2024-03-25T18:10:17.752 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.750+0000 7f15c7b7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:17.752 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.750+0000 7f15c7b7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:17.752 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.750+0000 7f15c7b7c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:10:17.752 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:17.750+0000 7f15c7b7c640 1 -- 172.21.15.156:0/3012496183 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15c00627b0 con 0x7f15c00a98f0 2024-03-25T18:10:18.064 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.063+0000 7f15b67fc640 1 -- 172.21.15.156:0/3012496183 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15bc001d00 con 0x7f15c00a98f0 2024-03-25T18:10:18.064 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.063+0000 7f15b67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:18.064 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.063+0000 7f15b67fc640 1 -- 172.21.15.156:0/3012496183 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f15bc004a00 con 0x7f15c00a98f0 2024-03-25T18:10:18.064 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.063+0000 7f15b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:18.064 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.063+0000 7f15b67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:18.065 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.063+0000 7f15b67fc640 10 monclient: dump: 2024-03-25T18:10:18.065 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:18.065 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:18.065 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:18.065 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:18.065 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:18.065 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:18.065 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:18.065 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:18.065 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:18.065 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:18.065 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.063+0000 7f15b7fff640 4 set_mon_vals no callback set 2024-03-25T18:10:18.066 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.064+0000 7f15b67fc640 1 -- 172.21.15.156:0/3012496183 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f15bc01aa50 con 0x7f15c00a98f0 2024-03-25T18:10:18.066 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.064+0000 7f15c5117640 10 monclient: get_auth_request con 0x7f15a406e060 auth_method 0 2024-03-25T18:10:18.066 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.065+0000 7f15b67fc640 10 client.4314.objecter ms_handle_connect 0x7f15a406e060 2024-03-25T18:10:18.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.069+0000 7f15b67fc640 1 -- 172.21.15.156:0/3012496183 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f15bc0547b0 con 0x7f15c00a98f0 2024-03-25T18:10:18.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.069+0000 7f15b67fc640 10 client.4314.objecter ms_dispatch 0x7f15c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:18.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.069+0000 7f15b67fc640 3 client.4314.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:18.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.069+0000 7f15b67fc640 3 client.4314.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:18.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.070+0000 7f15b67fc640 20 client.4314.objecter dump_active .. 0 homeless 2024-03-25T18:10:18.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.070+0000 7f15b67fc640 1 -- 172.21.15.156:0/3012496183 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f15bc0500f0 con 0x7f15c00a98f0 2024-03-25T18:10:18.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.070+0000 7f15b67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:18.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.070+0000 7f15b67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:18.294 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.293+0000 7f15c7b7c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:18.294 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.293+0000 7f15c7b7c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:18.294 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.293+0000 7f15c7b7c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:10:18.294 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.293+0000 7f15c7b7c640 1 -- 172.21.15.156:0/3012496183 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15c013ce70 con 0x7f15c00a98f0 2024-03-25T18:10:18.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.326+0000 7f15b67fc640 1 -- 172.21.15.156:0/3012496183 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f15bc0521f0 con 0x7f15c00a98f0 2024-03-25T18:10:18.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.326+0000 7f15b67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:18.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.326+0000 7f15b67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:18.327 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:10:18.327 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:18.331 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.329+0000 7f15c7b7c640 10 librados: watch_flush enter 2024-03-25T18:10:18.331 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.329+0000 7f15c7b7c640 10 librados: watch_flush exit 2024-03-25T18:10:18.331 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.329+0000 7f15c7b7c640 20 client.4314.objecter shutdown clearing up homeless session... 2024-03-25T18:10:18.331 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.329+0000 7f15c7b7c640 10 client.4314.objecter successfully canceled tick 2024-03-25T18:10:18.331 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.329+0000 7f15c7b7c640 1 -- 172.21.15.156:0/3012496183 >> v1:172.21.15.194:6800/2946427713 conn(0x7f15a406e060 legacy=0x7f15a4070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:18.331 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.329+0000 7f15c7b7c640 10 monclient: shutdown 2024-03-25T18:10:18.331 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.329+0000 7f15c7b7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:18.331 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.329+0000 7f15c7b7c640 1 -- 172.21.15.156:0/3012496183 >> v1:172.21.15.156:6789/0 conn(0x7f15c00a98f0 legacy=0x7f15c0138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:18.331 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.329+0000 7f15c7b7c640 1 -- 172.21.15.156:0/3012496183 shutdown_connections 2024-03-25T18:10:18.331 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.329+0000 7f15c7b7c640 1 -- 172.21.15.156:0/3012496183 >> 172.21.15.156:0/3012496183 conn(0x7f15c00b7ea0 msgr2=0x7f15c0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:18.331 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.329+0000 7f15c7b7c640 1 -- 172.21.15.156:0/3012496183 shutdown_connections 2024-03-25T18:10:18.331 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.329+0000 7f15c7b7c640 1 -- 172.21.15.156:0/3012496183 wait complete. 2024-03-25T18:10:18.332 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:18.329+0000 7f15c7b7c640 1 librados: shutdown 2024-03-25T18:10:18.343 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:25.149 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:25.264 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.264+0000 7fc5b0b02640 10 monclient: get_monmap_and_config 2024-03-25T18:10:25.264 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.264+0000 7fc5b0b02640 10 monclient: build_initial_monmap 2024-03-25T18:10:25.264 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.264+0000 7fc5b0b02640 10 monclient: monmap: 2024-03-25T18:10:25.265 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:10:25.265 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:25.265 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:10:25.265058+0000 2024-03-25T18:10:25.265 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:10:25.265058+0000 2024-03-25T18:10:25.265 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:25.265 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:25.265 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:10:25.265 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:10:25.265 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:10:25.265 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:25.266 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.265+0000 7fc5b0b02640 1 Processor -- start 2024-03-25T18:10:25.266 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.265+0000 7fc5b0b02640 1 -- start start 2024-03-25T18:10:25.266 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.265+0000 7fc5b0b02640 10 monclient: init 2024-03-25T18:10:25.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.265+0000 7fc5b0b02640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:25.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.265+0000 7fc5b0b02640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:10:25.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.265+0000 7fc5b0b02640 10 monclient(hunting): picked mon.noname-a con 0x7fc5ac06ea40 addr v1:172.21.15.156:6789/0 2024-03-25T18:10:25.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.265+0000 7fc5b0b02640 10 monclient(hunting): picked mon.noname-c con 0x7fc5ac071800 addr v1:172.21.15.194:6789/0 2024-03-25T18:10:25.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.265+0000 7fc5b0b02640 10 monclient(hunting): picked mon.noname-b con 0x7fc5ac071c40 addr v1:172.21.15.156:6790/0 2024-03-25T18:10:25.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.265+0000 7fc5b0b02640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5ac06f1f0 con 0x7fc5ac06ea40 2024-03-25T18:10:25.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.265+0000 7fc5b0b02640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5ac070d00 con 0x7fc5ac071800 2024-03-25T18:10:25.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.265+0000 7fc5b0b02640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5ac0720b0 con 0x7fc5ac071c40 2024-03-25T18:10:25.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.265+0000 7fc5b0b02640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:25.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.265+0000 7fc5b0b02640 10 monclient(hunting): authenticate will time out at 7106.576784s 2024-03-25T18:10:25.269 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.269+0000 7fc5a3fff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fc5ac071800 0x7fc5ac147070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:36990/0 (socket says 172.21.15.156:36990) 2024-03-25T18:10:25.269 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.269+0000 7fc5a3fff640 1 -- 172.21.15.156:0/2420606360 learned_addr learned my addr 172.21.15.156:0/2420606360 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:10:25.365 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.364+0000 7fc5a9d9b640 1 -- 172.21.15.156:0/2420606360 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 525606687 0 0) 0x7fc5ac06f1f0 con 0x7fc5ac06ea40 2024-03-25T18:10:25.365 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.365+0000 7fc5a9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:10:25.365 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.365+0000 7fc5a9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:25.365 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.365+0000 7fc5a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:25.365 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.365+0000 7fc5a9d9b640 10 monclient(hunting): my global_id is 4317 2024-03-25T18:10:25.365 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.365+0000 7fc5a9d9b640 1 -- 172.21.15.156:0/2420606360 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc598003520 con 0x7fc5ac06ea40 2024-03-25T18:10:25.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.380+0000 7fc5a9d9b640 1 -- 172.21.15.156:0/2420606360 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1324246649 0 0) 0x7fc598003520 con 0x7fc5ac06ea40 2024-03-25T18:10:25.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.380+0000 7fc5a9d9b640 1 -- 172.21.15.156:0/2420606360 >> v1:172.21.15.156:6790/0 conn(0x7fc5ac071c40 legacy=0x7fc5ac14a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:25.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.380+0000 7fc5a9d9b640 1 -- 172.21.15.156:0/2420606360 >> v1:172.21.15.194:6789/0 conn(0x7fc5ac071800 legacy=0x7fc5ac147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:25.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.380+0000 7fc5a9d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:10:25.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.380+0000 7fc5a9d9b640 1 monclient: found mon.noname-a 2024-03-25T18:10:25.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.380+0000 7fc5a9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:25.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.380+0000 7fc5a9d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:10:25.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.380+0000 7fc5a9d9b640 1 -- 172.21.15.156:0/2420606360 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ac072780 con 0x7fc5ac06ea40 2024-03-25T18:10:25.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.380+0000 7fc5a9d9b640 10 monclient: _finish_auth 0 2024-03-25T18:10:25.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.380+0000 7fc5a9d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:10:25.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.380+0000 7fc5a9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:25.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.380+0000 7fc5a9d9b640 1 -- 172.21.15.156:0/2420606360 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc59c001690 con 0x7fc5ac06ea40 2024-03-25T18:10:25.382 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.380+0000 7fc5a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:25.382 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.380+0000 7fc5a9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:25.382 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.380+0000 7fc5a9d9b640 10 monclient: dump: 2024-03-25T18:10:25.382 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:25.382 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:25.382 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:25.382 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:25.382 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:25.382 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:25.382 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:25.382 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:25.382 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:25.382 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:25.382 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.380+0000 7fc5b0b02640 5 monclient: authenticate success, global_id 4317 2024-03-25T18:10:25.382 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.380+0000 7fc5b0b02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:25.405 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.404+0000 7fc5a9d9b640 1 -- 172.21.15.156:0/2420606360 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc59c0017f0 con 0x7fc5ac06ea40 2024-03-25T18:10:25.405 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.404+0000 7fc5a9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:25.405 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.404+0000 7fc5a9d9b640 1 -- 172.21.15.156:0/2420606360 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc59c001c00 con 0x7fc5ac06ea40 2024-03-25T18:10:25.405 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.404+0000 7fc5b0b02640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:25.406 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.404+0000 7fc5b0b02640 4 set_mon_vals no callback set 2024-03-25T18:10:25.406 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.404+0000 7fc5a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:25.406 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.404+0000 7fc5a9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:25.406 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.404+0000 7fc5a9d9b640 10 monclient: dump: 2024-03-25T18:10:25.406 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:25.406 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:25.406 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:25.406 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:25.406 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:25.406 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:25.406 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:25.406 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:25.406 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:25.406 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:25.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.405+0000 7fc5b0b02640 10 monclient: shutdown 2024-03-25T18:10:25.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.405+0000 7fc5b0b02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:25.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.405+0000 7fc5b0b02640 1 -- 172.21.15.156:0/2420606360 >> v1:172.21.15.156:6789/0 conn(0x7fc5ac06ea40 legacy=0x7fc5ac13ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:25.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.405+0000 7fc5b0b02640 1 -- 172.21.15.156:0/2420606360 shutdown_connections 2024-03-25T18:10:25.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.405+0000 7fc5b0b02640 1 -- 172.21.15.156:0/2420606360 >> 172.21.15.156:0/2420606360 conn(0x7fc5ac06b630 msgr2=0x7fc5ac0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:25.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.405+0000 7fc5b0b02640 1 -- 172.21.15.156:0/2420606360 shutdown_connections 2024-03-25T18:10:25.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.405+0000 7fc5b0b02640 1 -- 172.21.15.156:0/2420606360 wait complete. 2024-03-25T18:10:25.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.406+0000 7fc5b0b02640 10 monclient: build_initial_monmap 2024-03-25T18:10:25.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.406+0000 7fc5b0b02640 10 monclient: monmap: 2024-03-25T18:10:25.408 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:10:25.408 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:25.409 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:10:25.409 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:10:25.409 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:25.409 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:10:25.409 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:10:25.409 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:10:25.409 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:10:25.409 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:25.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.407+0000 7fc5b0b02640 1 librados: starting msgr at 2024-03-25T18:10:25.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.407+0000 7fc5b0b02640 1 librados: starting objecter 2024-03-25T18:10:25.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.407+0000 7fc5b0b02640 1 Processor -- start 2024-03-25T18:10:25.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.407+0000 7fc5b0b02640 1 -- start start 2024-03-25T18:10:25.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.407+0000 7fc5b0b02640 1 librados: setting wanted keys 2024-03-25T18:10:25.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.407+0000 7fc5b0b02640 1 librados: calling monclient init 2024-03-25T18:10:25.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.407+0000 7fc5b0b02640 10 monclient: init 2024-03-25T18:10:25.410 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.407+0000 7fc5b0b02640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:25.410 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.407+0000 7fc5b0b02640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:10:25.410 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.407+0000 7fc5b0b02640 10 monclient(hunting): picked mon.noname-b con 0x7fc5ac06ea40 addr v1:172.21.15.194:6789/0 2024-03-25T18:10:25.410 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.408+0000 7fc5b0b02640 10 monclient(hunting): picked mon.noname-c con 0x7fc5ac071800 addr v1:172.21.15.156:6790/0 2024-03-25T18:10:25.410 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.408+0000 7fc5b0b02640 10 monclient(hunting): picked mon.noname-a con 0x7fc5ac071c40 addr v1:172.21.15.156:6789/0 2024-03-25T18:10:25.410 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.408+0000 7fc5b0b02640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5ac1dae70 con 0x7fc5ac071c40 2024-03-25T18:10:25.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.408+0000 7fc5b0b02640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5ac0ad6e0 con 0x7fc5ac06ea40 2024-03-25T18:10:25.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.408+0000 7fc5b0b02640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5ac0aadb0 con 0x7fc5ac071800 2024-03-25T18:10:25.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.408+0000 7fc5b0b02640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:25.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.408+0000 7fc5b0b02640 10 monclient(hunting): authenticate will time out at 7106.718916s 2024-03-25T18:10:25.423 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.422+0000 7fc5aa59c640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fc5ac06ea40 0x7fc5ac1db390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:37046/0 (socket says 172.21.15.156:37046) 2024-03-25T18:10:25.423 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.422+0000 7fc5aa59c640 1 -- 172.21.15.156:0/94351731 learned_addr learned my addr 172.21.15.156:0/94351731 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:10:25.424 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.424+0000 7fc5a2ffd640 10 client.?.objecter ms_handle_connect 0x7fc5ac06ea40 2024-03-25T18:10:25.424 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.424+0000 7fc5a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:25.426 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.426+0000 7fc5a2ffd640 10 client.?.objecter ms_handle_connect 0x7fc5ac071800 2024-03-25T18:10:25.426 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.426+0000 7fc5a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:25.429 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.428+0000 7fc5a2ffd640 10 client.?.objecter ms_handle_connect 0x7fc5ac071c40 2024-03-25T18:10:25.429 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.428+0000 7fc5a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:25.436 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.435+0000 7fc5a2ffd640 1 -- 172.21.15.156:0/94351731 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074757697 0 0) 0x7fc5ac0aadb0 con 0x7fc5ac071800 2024-03-25T18:10:25.436 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.435+0000 7fc5a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:25.436 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.435+0000 7fc5a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:25.436 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.435+0000 7fc5a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:25.436 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.435+0000 7fc5a2ffd640 10 monclient(hunting): my global_id is 4328 2024-03-25T18:10:25.437 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.435+0000 7fc5a2ffd640 1 -- 172.21.15.156:0/94351731 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc58c004eb0 con 0x7fc5ac071800 2024-03-25T18:10:25.437 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.436+0000 7fc5a2ffd640 1 -- 172.21.15.156:0/94351731 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274246239 0 0) 0x7fc5ac0ad6e0 con 0x7fc5ac06ea40 2024-03-25T18:10:25.437 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.436+0000 7fc5a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:25.437 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.436+0000 7fc5a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:25.437 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.436+0000 7fc5a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:25.437 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.436+0000 7fc5a2ffd640 10 monclient(hunting): my global_id is 4315 2024-03-25T18:10:25.437 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.436+0000 7fc5a2ffd640 1 -- 172.21.15.156:0/94351731 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5ac0aadb0 con 0x7fc5ac06ea40 2024-03-25T18:10:25.438 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.437+0000 7fc5a2ffd640 1 -- 172.21.15.156:0/94351731 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 329051842 0 0) 0x7fc5ac1dae70 con 0x7fc5ac071c40 2024-03-25T18:10:25.438 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.437+0000 7fc5a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:25.438 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.437+0000 7fc5a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:25.438 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.437+0000 7fc5a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:25.439 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.437+0000 7fc5a2ffd640 10 monclient(hunting): my global_id is 4320 2024-03-25T18:10:25.439 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.437+0000 7fc5a2ffd640 1 -- 172.21.15.156:0/94351731 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5ac0ad6e0 con 0x7fc5ac071c40 2024-03-25T18:10:25.456 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.455+0000 7fc5a2ffd640 1 -- 172.21.15.156:0/94351731 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3941570994 0 0) 0x7fc58c004eb0 con 0x7fc5ac071800 2024-03-25T18:10:25.456 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.455+0000 7fc5a2ffd640 1 -- 172.21.15.156:0/94351731 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5ac1dae70 con 0x7fc5ac071800 2024-03-25T18:10:25.456 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.455+0000 7fc5a2ffd640 1 -- 172.21.15.156:0/94351731 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc5940018a0 con 0x7fc5ac071800 2024-03-25T18:10:25.456 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.455+0000 7fc5a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:25.456 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.455+0000 7fc5a2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:25.456 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.455+0000 7fc5a2ffd640 10 monclient(hunting): dump: 2024-03-25T18:10:25.456 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:25.456 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:25.456 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:25.456 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:25.456 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:25.457 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:25.457 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:25.457 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:25.457 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:25.457 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:25.457 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.455+0000 7fc5a2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:25.457 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.455+0000 7fc5a2ffd640 1 -- 172.21.15.156:0/94351731 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1363700247 0 0) 0x7fc5ac0ad6e0 con 0x7fc5ac071c40 2024-03-25T18:10:25.457 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.455+0000 7fc5a2ffd640 1 -- 172.21.15.156:0/94351731 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc58c004eb0 con 0x7fc5ac071c40 2024-03-25T18:10:25.457 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.455+0000 7fc5a2ffd640 1 -- 172.21.15.156:0/94351731 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc5a40019e0 con 0x7fc5ac071c40 2024-03-25T18:10:25.457 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.455+0000 7fc5a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:25.457 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.455+0000 7fc5a2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:25.457 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.455+0000 7fc5a2ffd640 10 monclient(hunting): dump: 2024-03-25T18:10:25.457 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:25.457 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:25.457 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:25.457 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:25.458 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:25.458 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:25.458 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:25.458 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:25.458 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:25.458 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:25.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.459+0000 7fc5a2ffd640 1 -- 172.21.15.156:0/94351731 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 244770298 0 0) 0x7fc5ac0aadb0 con 0x7fc5ac06ea40 2024-03-25T18:10:25.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.459+0000 7fc5a2ffd640 1 -- 172.21.15.156:0/94351731 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5ac0ad6e0 con 0x7fc5ac06ea40 2024-03-25T18:10:25.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.459+0000 7fc5a2ffd640 1 -- 172.21.15.156:0/94351731 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc59c001690 con 0x7fc5ac06ea40 2024-03-25T18:10:25.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.459+0000 7fc5a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:25.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.459+0000 7fc5a2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:25.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.459+0000 7fc5a2ffd640 10 monclient(hunting): dump: 2024-03-25T18:10:25.460 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:25.460 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:25.460 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:25.461 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:25.461 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:25.461 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:25.461 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:25.461 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:25.461 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:25.461 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:25.468 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.467+0000 7fc5a2ffd640 1 -- 172.21.15.156:0/94351731 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1464000390 0 0) 0x7fc5ac0ad6e0 con 0x7fc5ac06ea40 2024-03-25T18:10:25.468 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.467+0000 7fc5a2ffd640 1 -- 172.21.15.156:0/94351731 >> v1:172.21.15.156:6790/0 conn(0x7fc5ac071800 legacy=0x7fc5ac13cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:25.468 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.467+0000 7fc5a2ffd640 1 -- 172.21.15.156:0/94351731 >> v1:172.21.15.156:6789/0 conn(0x7fc5ac071c40 legacy=0x7fc5ac1e3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:25.468 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.467+0000 7fc5a2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:10:25.468 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.467+0000 7fc5a2ffd640 1 monclient: found mon.b 2024-03-25T18:10:25.468 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.467+0000 7fc5a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:25.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.467+0000 7fc5a2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:10:25.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.467+0000 7fc5a2ffd640 1 -- 172.21.15.156:0/94351731 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ac0aaf40 con 0x7fc5ac06ea40 2024-03-25T18:10:25.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.467+0000 7fc5a2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:10:25.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.467+0000 7fc5a2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:10:25.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.467+0000 7fc5a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:25.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.467+0000 7fc5b0b02640 5 monclient: authenticate success, global_id 4315 2024-03-25T18:10:25.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.467+0000 7fc5b0b02640 10 monclient: _renew_subs 2024-03-25T18:10:25.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.467+0000 7fc5b0b02640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:10:25.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.467+0000 7fc5b0b02640 1 -- 172.21.15.156:0/94351731 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5ac11ff40 con 0x7fc5ac06ea40 2024-03-25T18:10:25.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.467+0000 7fc5b0b02640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:25.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.467+0000 7fc5b0b02640 10 monclient: _renew_subs 2024-03-25T18:10:25.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.467+0000 7fc5b0b02640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:10:25.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.467+0000 7fc5b0b02640 1 -- 172.21.15.156:0/94351731 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5ac120d00 con 0x7fc5ac06ea40 2024-03-25T18:10:25.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.467+0000 7fc5b0b02640 1 librados: init done 2024-03-25T18:10:25.470 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.468+0000 7fc5b0b02640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:25.470 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.468+0000 7fc5b0b02640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:25.470 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.468+0000 7fc5b0b02640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:10:25.470 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.468+0000 7fc5b0b02640 1 -- 172.21.15.156:0/94351731 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5ac0627b0 con 0x7fc5ac06ea40 2024-03-25T18:10:25.742 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.740+0000 7fc5a2ffd640 1 -- 172.21.15.156:0/94351731 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc59c0038a0 con 0x7fc5ac06ea40 2024-03-25T18:10:25.742 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.740+0000 7fc5a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:25.742 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.740+0000 7fc5a2ffd640 1 -- 172.21.15.156:0/94351731 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc59c003c70 con 0x7fc5ac06ea40 2024-03-25T18:10:25.742 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.740+0000 7fc5a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:25.742 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.741+0000 7fc5a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:25.742 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.740+0000 7fc5a959a640 4 set_mon_vals no callback set 2024-03-25T18:10:25.742 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.741+0000 7fc5a2ffd640 10 monclient: dump: 2024-03-25T18:10:25.742 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:25.742 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:25.742 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:25.742 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:25.743 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:25.743 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:25.743 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:25.743 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:25.743 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:25.743 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:25.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.741+0000 7fc5a2ffd640 1 -- 172.21.15.156:0/94351731 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7fc59c003e90 con 0x7fc5ac06ea40 2024-03-25T18:10:25.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.742+0000 7fc5a3fff640 10 monclient: get_auth_request con 0x7fc58c06e160 auth_method 0 2024-03-25T18:10:25.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.742+0000 7fc5a2ffd640 10 client.4315.objecter ms_handle_connect 0x7fc58c06e160 2024-03-25T18:10:25.748 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.747+0000 7fc5a2ffd640 1 -- 172.21.15.156:0/94351731 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7fc59c0548e0 con 0x7fc5ac06ea40 2024-03-25T18:10:25.748 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.747+0000 7fc5a2ffd640 10 client.4315.objecter ms_dispatch 0x7fc5ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:25.748 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.747+0000 7fc5a2ffd640 3 client.4315.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:25.748 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.747+0000 7fc5a2ffd640 3 client.4315.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:25.748 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.747+0000 7fc5a2ffd640 20 client.4315.objecter dump_active .. 0 homeless 2024-03-25T18:10:25.749 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.748+0000 7fc5a2ffd640 1 -- 172.21.15.156:0/94351731 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc59c0503f0 con 0x7fc5ac06ea40 2024-03-25T18:10:25.749 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.748+0000 7fc5a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:25.749 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:25.748+0000 7fc5a2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:26.008 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:26.008+0000 7fc5b0b02640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:26.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:26.008+0000 7fc5b0b02640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:26.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:26.008+0000 7fc5b0b02640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:10:26.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:26.008+0000 7fc5b0b02640 1 -- 172.21.15.156:0/94351731 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5ac1db0b0 con 0x7fc5ac06ea40 2024-03-25T18:10:26.040 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:26.039+0000 7fc5a2ffd640 1 -- 172.21.15.156:0/94351731 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7fc5ac1db0b0 con 0x7fc5ac06ea40 2024-03-25T18:10:26.040 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:26.039+0000 7fc5a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:26.040 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:26.039+0000 7fc5a2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:26.040 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:10:26.040 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:26.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:26.041+0000 7fc5b0b02640 10 librados: watch_flush enter 2024-03-25T18:10:26.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:26.041+0000 7fc5b0b02640 10 librados: watch_flush exit 2024-03-25T18:10:26.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:26.041+0000 7fc5b0b02640 20 client.4315.objecter shutdown clearing up homeless session... 2024-03-25T18:10:26.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:26.041+0000 7fc5b0b02640 10 client.4315.objecter successfully canceled tick 2024-03-25T18:10:26.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:26.041+0000 7fc5b0b02640 1 -- 172.21.15.156:0/94351731 >> v1:172.21.15.194:6800/2946427713 conn(0x7fc58c06e160 legacy=0x7fc58c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:26.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:26.041+0000 7fc5b0b02640 10 monclient: shutdown 2024-03-25T18:10:26.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:26.041+0000 7fc5b0b02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:26.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:26.041+0000 7fc5b0b02640 1 -- 172.21.15.156:0/94351731 >> v1:172.21.15.194:6789/0 conn(0x7fc5ac06ea40 legacy=0x7fc5ac1db390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:26.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:26.041+0000 7fc5b0b02640 1 -- 172.21.15.156:0/94351731 shutdown_connections 2024-03-25T18:10:26.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:26.041+0000 7fc5b0b02640 1 -- 172.21.15.156:0/94351731 >> 172.21.15.156:0/94351731 conn(0x7fc5ac06b630 msgr2=0x7fc5ac0af720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:26.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:26.041+0000 7fc5b0b02640 1 -- 172.21.15.156:0/94351731 shutdown_connections 2024-03-25T18:10:26.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:26.041+0000 7fc5b0b02640 1 -- 172.21.15.156:0/94351731 wait complete. 2024-03-25T18:10:26.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:26.041+0000 7fc5b0b02640 1 librados: shutdown 2024-03-25T18:10:26.054 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:32.860 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:32.974 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:32.974+0000 7f6e91863640 10 monclient: get_monmap_and_config 2024-03-25T18:10:32.975 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:32.974+0000 7f6e91863640 10 monclient: build_initial_monmap 2024-03-25T18:10:32.975 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:32.974+0000 7f6e91863640 10 monclient: monmap: 2024-03-25T18:10:32.975 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:10:32.975 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:32.975 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:10:32.975138+0000 2024-03-25T18:10:32.975 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:10:32.975138+0000 2024-03-25T18:10:32.975 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:32.975 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:32.975 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:10:32.975 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:10:32.975 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:10:32.976 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:32.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:32.974+0000 7f6e91863640 1 Processor -- start 2024-03-25T18:10:32.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:32.975+0000 7f6e91863640 1 -- start start 2024-03-25T18:10:32.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:32.975+0000 7f6e91863640 10 monclient: init 2024-03-25T18:10:32.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:32.975+0000 7f6e91863640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:32.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:32.975+0000 7f6e91863640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:10:32.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:32.975+0000 7f6e91863640 10 monclient(hunting): picked mon.noname-b con 0x7f6e8c13fc60 addr v1:172.21.15.156:6790/0 2024-03-25T18:10:32.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:32.975+0000 7f6e91863640 10 monclient(hunting): picked mon.noname-c con 0x7f6e8c070570 addr v1:172.21.15.194:6789/0 2024-03-25T18:10:32.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:32.975+0000 7f6e91863640 10 monclient(hunting): picked mon.noname-a con 0x7f6e8c0727a0 addr v1:172.21.15.156:6789/0 2024-03-25T18:10:32.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:32.975+0000 7f6e91863640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e8c13c9f0 con 0x7f6e8c0727a0 2024-03-25T18:10:32.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:32.975+0000 7f6e91863640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e8c06fac0 con 0x7f6e8c070570 2024-03-25T18:10:32.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:32.975+0000 7f6e91863640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e8c0715d0 con 0x7f6e8c13fc60 2024-03-25T18:10:32.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:32.975+0000 7f6e91863640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:32.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:32.975+0000 7f6e91863640 10 monclient(hunting): authenticate will time out at 7114.286455s 2024-03-25T18:10:32.984 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:32.983+0000 7f6e8affd640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f6e8c13fc60 0x7f6e8c06f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:52904/0 (socket says 172.21.15.156:52904) 2024-03-25T18:10:32.984 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:32.983+0000 7f6e8affd640 1 -- 172.21.15.156:0/3767615994 learned_addr learned my addr 172.21.15.156:0/3767615994 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:10:33.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.003+0000 7f6e8a7fc640 1 -- 172.21.15.156:0/3767615994 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 336501137 0 0) 0x7f6e8c06fac0 con 0x7f6e8c070570 2024-03-25T18:10:33.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.003+0000 7f6e8a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:10:33.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.003+0000 7f6e8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:33.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.003+0000 7f6e8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:33.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.003+0000 7f6e8a7fc640 10 monclient(hunting): my global_id is 4318 2024-03-25T18:10:33.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.003+0000 7f6e8a7fc640 1 -- 172.21.15.156:0/3767615994 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e78005730 con 0x7f6e8c070570 2024-03-25T18:10:33.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.009+0000 7f6e8a7fc640 1 -- 172.21.15.156:0/3767615994 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2175649112 0 0) 0x7f6e8c0715d0 con 0x7f6e8c13fc60 2024-03-25T18:10:33.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.009+0000 7f6e8a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:10:33.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.009+0000 7f6e8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:33.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.009+0000 7f6e8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:33.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.009+0000 7f6e8a7fc640 10 monclient(hunting): my global_id is 4331 2024-03-25T18:10:33.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.009+0000 7f6e8a7fc640 1 -- 172.21.15.156:0/3767615994 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e8c06fac0 con 0x7f6e8c13fc60 2024-03-25T18:10:33.017 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.016+0000 7f6e8a7fc640 1 -- 172.21.15.156:0/3767615994 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2495838819 0 0) 0x7f6e78005730 con 0x7f6e8c070570 2024-03-25T18:10:33.017 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.016+0000 7f6e8a7fc640 1 -- 172.21.15.156:0/3767615994 >> v1:172.21.15.156:6790/0 conn(0x7f6e8c13fc60 legacy=0x7f6e8c06f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:33.017 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.016+0000 7f6e8a7fc640 1 -- 172.21.15.156:0/3767615994 >> v1:172.21.15.156:6789/0 conn(0x7f6e8c0727a0 legacy=0x7f6e8c14a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:33.017 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.016+0000 7f6e8a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:10:33.017 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.016+0000 7f6e8a7fc640 1 monclient: found mon.noname-c 2024-03-25T18:10:33.017 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.016+0000 7f6e8a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:33.017 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.016+0000 7f6e8a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:10:33.018 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.016+0000 7f6e8a7fc640 1 -- 172.21.15.156:0/3767615994 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e8c0718b0 con 0x7f6e8c070570 2024-03-25T18:10:33.018 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.016+0000 7f6e8a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:10:33.018 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.016+0000 7f6e8a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:10:33.018 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.016+0000 7f6e8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:33.018 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.016+0000 7f6e8a7fc640 1 -- 172.21.15.156:0/3767615994 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f6e74001690 con 0x7f6e8c070570 2024-03-25T18:10:33.018 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.016+0000 7f6e8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:33.018 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.016+0000 7f6e8a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:33.018 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.016+0000 7f6e8a7fc640 10 monclient: dump: 2024-03-25T18:10:33.018 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:33.018 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:33.018 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:33.018 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:33.018 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:33.018 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:33.018 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:33.018 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:33.019 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:33.019 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:33.019 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.016+0000 7f6e91863640 5 monclient: authenticate success, global_id 4318 2024-03-25T18:10:33.019 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.016+0000 7f6e91863640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:33.033 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.032+0000 7f6e8a7fc640 1 -- 172.21.15.156:0/3767615994 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e740017f0 con 0x7f6e8c070570 2024-03-25T18:10:33.033 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.032+0000 7f6e8a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:33.033 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.032+0000 7f6e8a7fc640 1 -- 172.21.15.156:0/3767615994 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f6e74001c00 con 0x7f6e8c070570 2024-03-25T18:10:33.033 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.032+0000 7f6e8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:33.033 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.032+0000 7f6e8a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:33.033 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.032+0000 7f6e8a7fc640 10 monclient: dump: 2024-03-25T18:10:33.033 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:33.034 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:33.034 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:33.034 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:33.034 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:33.034 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:33.034 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:33.034 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:33.034 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:33.034 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:33.034 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.032+0000 7f6e91863640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:33.034 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.032+0000 7f6e91863640 4 set_mon_vals no callback set 2024-03-25T18:10:33.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.033+0000 7f6e91863640 10 monclient: shutdown 2024-03-25T18:10:33.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.033+0000 7f6e91863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:33.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.033+0000 7f6e91863640 1 -- 172.21.15.156:0/3767615994 >> v1:172.21.15.194:6789/0 conn(0x7f6e8c070570 legacy=0x7f6e8c072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:33.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.033+0000 7f6e91863640 1 -- 172.21.15.156:0/3767615994 shutdown_connections 2024-03-25T18:10:33.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.033+0000 7f6e91863640 1 -- 172.21.15.156:0/3767615994 >> 172.21.15.156:0/3767615994 conn(0x7f6e8c0b4a80 msgr2=0x7f6e8c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:33.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.033+0000 7f6e91863640 1 -- 172.21.15.156:0/3767615994 shutdown_connections 2024-03-25T18:10:33.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.033+0000 7f6e91863640 1 -- 172.21.15.156:0/3767615994 wait complete. 2024-03-25T18:10:33.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.034+0000 7f6e91863640 10 monclient: build_initial_monmap 2024-03-25T18:10:33.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.034+0000 7f6e91863640 10 monclient: monmap: 2024-03-25T18:10:33.037 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:10:33.037 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:33.037 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:10:33.037 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:10:33.037 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:33.037 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:10:33.038 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:10:33.038 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:10:33.038 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:10:33.038 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:33.038 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.035+0000 7f6e91863640 1 librados: starting msgr at 2024-03-25T18:10:33.038 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.035+0000 7f6e91863640 1 librados: starting objecter 2024-03-25T18:10:33.038 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.035+0000 7f6e91863640 1 Processor -- start 2024-03-25T18:10:33.038 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.035+0000 7f6e91863640 1 -- start start 2024-03-25T18:10:33.038 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.035+0000 7f6e91863640 1 librados: setting wanted keys 2024-03-25T18:10:33.038 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.035+0000 7f6e91863640 1 librados: calling monclient init 2024-03-25T18:10:33.038 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.035+0000 7f6e91863640 10 monclient: init 2024-03-25T18:10:33.038 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.035+0000 7f6e91863640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:33.038 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.035+0000 7f6e91863640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:10:33.038 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.035+0000 7f6e91863640 10 monclient(hunting): picked mon.noname-c con 0x7f6e8c070570 addr v1:172.21.15.156:6790/0 2024-03-25T18:10:33.039 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.035+0000 7f6e91863640 10 monclient(hunting): picked mon.noname-a con 0x7f6e8c0727a0 addr v1:172.21.15.156:6789/0 2024-03-25T18:10:33.039 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.035+0000 7f6e91863640 10 monclient(hunting): picked mon.noname-b con 0x7f6e8c13fc60 addr v1:172.21.15.194:6789/0 2024-03-25T18:10:33.039 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.035+0000 7f6e91863640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e8c138ab0 con 0x7f6e8c0727a0 2024-03-25T18:10:33.039 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.035+0000 7f6e91863640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e8c1d4620 con 0x7f6e8c13fc60 2024-03-25T18:10:33.039 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.035+0000 7f6e91863640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e8c138d00 con 0x7f6e8c070570 2024-03-25T18:10:33.039 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.036+0000 7f6e91863640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:33.039 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.036+0000 7f6e91863640 10 monclient(hunting): authenticate will time out at 7114.346798s 2024-03-25T18:10:33.045 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.045+0000 7f6e8b7fe640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f6e8c13fc60 0x7f6e8c132ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:38026/0 (socket says 172.21.15.156:38026) 2024-03-25T18:10:33.046 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.045+0000 7f6e8b7fe640 1 -- 172.21.15.156:0/1545008282 learned_addr learned my addr 172.21.15.156:0/1545008282 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:10:33.046 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.046+0000 7f6e88ff9640 10 client.?.objecter ms_handle_connect 0x7f6e8c070570 2024-03-25T18:10:33.047 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.046+0000 7f6e88ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:33.047 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.047+0000 7f6e88ff9640 10 client.?.objecter ms_handle_connect 0x7f6e8c13fc60 2024-03-25T18:10:33.047 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.047+0000 7f6e88ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:33.050 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.049+0000 7f6e88ff9640 10 client.?.objecter ms_handle_connect 0x7f6e8c0727a0 2024-03-25T18:10:33.050 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.049+0000 7f6e88ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:33.059 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.059+0000 7f6e88ff9640 1 -- 172.21.15.156:0/1545008282 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2800562831 0 0) 0x7f6e8c138d00 con 0x7f6e8c070570 2024-03-25T18:10:33.060 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.059+0000 7f6e88ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:10:33.060 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.059+0000 7f6e88ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:33.060 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.059+0000 7f6e88ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:33.060 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.059+0000 7f6e88ff9640 10 monclient(hunting): my global_id is 4334 2024-03-25T18:10:33.060 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.059+0000 7f6e88ff9640 1 -- 172.21.15.156:0/1545008282 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e6c004eb0 con 0x7f6e8c070570 2024-03-25T18:10:33.062 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.061+0000 7f6e88ff9640 1 -- 172.21.15.156:0/1545008282 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3014763074 0 0) 0x7f6e8c1d4620 con 0x7f6e8c13fc60 2024-03-25T18:10:33.062 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.061+0000 7f6e88ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:10:33.062 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.061+0000 7f6e88ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:33.062 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.061+0000 7f6e88ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:33.062 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.061+0000 7f6e88ff9640 10 monclient(hunting): my global_id is 4321 2024-03-25T18:10:33.063 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.061+0000 7f6e88ff9640 1 -- 172.21.15.156:0/1545008282 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e8c138d00 con 0x7f6e8c13fc60 2024-03-25T18:10:33.065 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.065+0000 7f6e88ff9640 1 -- 172.21.15.156:0/1545008282 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802309193 0 0) 0x7f6e8c138ab0 con 0x7f6e8c0727a0 2024-03-25T18:10:33.065 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.065+0000 7f6e88ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:10:33.065 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.065+0000 7f6e88ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:33.066 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.065+0000 7f6e88ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:33.066 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.065+0000 7f6e88ff9640 10 monclient(hunting): my global_id is 4326 2024-03-25T18:10:33.066 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.065+0000 7f6e88ff9640 1 -- 172.21.15.156:0/1545008282 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e8c1d4620 con 0x7f6e8c0727a0 2024-03-25T18:10:33.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.079+0000 7f6e88ff9640 1 -- 172.21.15.156:0/1545008282 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2200542257 0 0) 0x7f6e8c1d4620 con 0x7f6e8c0727a0 2024-03-25T18:10:33.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.079+0000 7f6e88ff9640 1 -- 172.21.15.156:0/1545008282 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e8c138ab0 con 0x7f6e8c0727a0 2024-03-25T18:10:33.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.079+0000 7f6e88ff9640 1 -- 172.21.15.156:0/1545008282 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f6e74001b00 con 0x7f6e8c0727a0 2024-03-25T18:10:33.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.079+0000 7f6e88ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:33.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.079+0000 7f6e88ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:33.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.079+0000 7f6e88ff9640 10 monclient(hunting): dump: 2024-03-25T18:10:33.080 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:33.080 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:33.081 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:33.081 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:33.081 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:33.081 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:33.081 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:33.081 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:33.081 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:33.081 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:33.081 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.079+0000 7f6e88ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:33.087 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.086+0000 7f6e88ff9640 1 -- 172.21.15.156:0/1545008282 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1868437794 0 0) 0x7f6e6c004eb0 con 0x7f6e8c070570 2024-03-25T18:10:33.087 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.086+0000 7f6e88ff9640 1 -- 172.21.15.156:0/1545008282 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e8c1d4620 con 0x7f6e8c070570 2024-03-25T18:10:33.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.086+0000 7f6e88ff9640 1 -- 172.21.15.156:0/1545008282 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f6e7c001890 con 0x7f6e8c070570 2024-03-25T18:10:33.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.086+0000 7f6e88ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:33.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.086+0000 7f6e88ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:33.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.086+0000 7f6e88ff9640 10 monclient(hunting): dump: 2024-03-25T18:10:33.088 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:33.088 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:33.088 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:33.088 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:33.088 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:33.088 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:33.088 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:33.088 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:33.088 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:33.088 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:33.091 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.091+0000 7f6e88ff9640 1 -- 172.21.15.156:0/1545008282 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1699443390 0 0) 0x7f6e8c138d00 con 0x7f6e8c13fc60 2024-03-25T18:10:33.092 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.091+0000 7f6e88ff9640 1 -- 172.21.15.156:0/1545008282 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e6c004eb0 con 0x7f6e8c13fc60 2024-03-25T18:10:33.092 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.091+0000 7f6e88ff9640 1 -- 172.21.15.156:0/1545008282 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f6e84001c80 con 0x7f6e8c13fc60 2024-03-25T18:10:33.092 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.091+0000 7f6e88ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:33.092 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.091+0000 7f6e88ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:33.092 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.091+0000 7f6e88ff9640 10 monclient(hunting): dump: 2024-03-25T18:10:33.092 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:33.092 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:33.092 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:33.092 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:33.092 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:33.092 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:33.092 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:33.092 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:33.093 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:33.093 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:33.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.096+0000 7f6e88ff9640 1 -- 172.21.15.156:0/1545008282 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2469857009 0 0) 0x7f6e8c138ab0 con 0x7f6e8c0727a0 2024-03-25T18:10:33.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.096+0000 7f6e88ff9640 1 -- 172.21.15.156:0/1545008282 >> v1:172.21.15.156:6790/0 conn(0x7f6e8c070570 legacy=0x7f6e8c142710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:33.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.096+0000 7f6e88ff9640 1 -- 172.21.15.156:0/1545008282 >> v1:172.21.15.194:6789/0 conn(0x7f6e8c13fc60 legacy=0x7f6e8c132ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:33.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.096+0000 7f6e88ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:10:33.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.096+0000 7f6e88ff9640 1 monclient: found mon.a 2024-03-25T18:10:33.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.096+0000 7f6e88ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:33.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.096+0000 7f6e88ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:10:33.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.096+0000 7f6e88ff9640 1 -- 172.21.15.156:0/1545008282 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e8c1d3710 con 0x7f6e8c0727a0 2024-03-25T18:10:33.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.096+0000 7f6e88ff9640 10 monclient: _finish_auth 0 2024-03-25T18:10:33.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.096+0000 7f6e88ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:10:33.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.096+0000 7f6e88ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:33.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.096+0000 7f6e91863640 5 monclient: authenticate success, global_id 4326 2024-03-25T18:10:33.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.096+0000 7f6e91863640 10 monclient: _renew_subs 2024-03-25T18:10:33.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.096+0000 7f6e91863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:10:33.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.096+0000 7f6e91863640 1 -- 172.21.15.156:0/1545008282 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e8c142fd0 con 0x7f6e8c0727a0 2024-03-25T18:10:33.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.096+0000 7f6e91863640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:33.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.096+0000 7f6e91863640 10 monclient: _renew_subs 2024-03-25T18:10:33.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.096+0000 7f6e91863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:10:33.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.096+0000 7f6e91863640 1 -- 172.21.15.156:0/1545008282 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e8c1337c0 con 0x7f6e8c0727a0 2024-03-25T18:10:33.099 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.096+0000 7f6e91863640 1 librados: init done 2024-03-25T18:10:33.099 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.097+0000 7f6e91863640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:33.099 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.097+0000 7f6e91863640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:33.099 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.097+0000 7f6e91863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:10:33.099 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.097+0000 7f6e91863640 1 -- 172.21.15.156:0/1545008282 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e8c0627b0 con 0x7f6e8c0727a0 2024-03-25T18:10:33.396 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.395+0000 7f6e88ff9640 1 -- 172.21.15.156:0/1545008282 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e74001c60 con 0x7f6e8c0727a0 2024-03-25T18:10:33.396 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.395+0000 7f6e88ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:33.396 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.395+0000 7f6e88ff9640 1 -- 172.21.15.156:0/1545008282 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f6e74003a40 con 0x7f6e8c0727a0 2024-03-25T18:10:33.396 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.395+0000 7f6e88ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:33.396 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.395+0000 7f6e88ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:33.396 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.395+0000 7f6e88ff9640 10 monclient: dump: 2024-03-25T18:10:33.396 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:33.396 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:33.396 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:33.396 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:33.396 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:33.397 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:33.397 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:33.397 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:33.397 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:33.397 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:33.397 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.395+0000 7f6e89ffb640 4 set_mon_vals no callback set 2024-03-25T18:10:33.397 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.396+0000 7f6e88ff9640 1 -- 172.21.15.156:0/1545008282 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f6e74019a10 con 0x7f6e8c0727a0 2024-03-25T18:10:33.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.396+0000 7f6e8affd640 10 monclient: get_auth_request con 0x7f6e6c06e0a0 auth_method 0 2024-03-25T18:10:33.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.396+0000 7f6e88ff9640 10 client.4326.objecter ms_handle_connect 0x7f6e6c06e0a0 2024-03-25T18:10:33.405 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.404+0000 7f6e88ff9640 1 -- 172.21.15.156:0/1545008282 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f6e74053670 con 0x7f6e8c0727a0 2024-03-25T18:10:33.405 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.404+0000 7f6e88ff9640 10 client.4326.objecter ms_dispatch 0x7f6e8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:33.405 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.404+0000 7f6e88ff9640 3 client.4326.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:33.405 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.404+0000 7f6e88ff9640 3 client.4326.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:33.405 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.404+0000 7f6e88ff9640 20 client.4326.objecter dump_active .. 0 homeless 2024-03-25T18:10:33.406 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.404+0000 7f6e88ff9640 1 -- 172.21.15.156:0/1545008282 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6e7404efb0 con 0x7f6e8c0727a0 2024-03-25T18:10:33.406 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.404+0000 7f6e88ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:33.406 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.404+0000 7f6e88ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:33.650 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.649+0000 7f6e91863640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:33.650 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.649+0000 7f6e91863640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:33.650 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.649+0000 7f6e91863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:10:33.650 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.649+0000 7f6e91863640 1 -- 172.21.15.156:0/1545008282 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e8c0af160 con 0x7f6e8c0727a0 2024-03-25T18:10:33.670 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.669+0000 7f6e88ff9640 1 -- 172.21.15.156:0/1545008282 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f6e740510b0 con 0x7f6e8c0727a0 2024-03-25T18:10:33.670 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.669+0000 7f6e88ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:33.670 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.669+0000 7f6e88ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:33.671 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:10:33.671 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:33.673 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.672+0000 7f6e91863640 10 librados: watch_flush enter 2024-03-25T18:10:33.673 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.672+0000 7f6e91863640 10 librados: watch_flush exit 2024-03-25T18:10:33.673 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.672+0000 7f6e91863640 20 client.4326.objecter shutdown clearing up homeless session... 2024-03-25T18:10:33.673 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.672+0000 7f6e91863640 10 client.4326.objecter successfully canceled tick 2024-03-25T18:10:33.673 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.672+0000 7f6e91863640 1 -- 172.21.15.156:0/1545008282 >> v1:172.21.15.194:6800/2946427713 conn(0x7f6e6c06e0a0 legacy=0x7f6e6c070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:33.674 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.672+0000 7f6e91863640 10 monclient: shutdown 2024-03-25T18:10:33.674 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.672+0000 7f6e91863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:33.674 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.672+0000 7f6e91863640 1 -- 172.21.15.156:0/1545008282 >> v1:172.21.15.156:6789/0 conn(0x7f6e8c0727a0 legacy=0x7f6e8c1d3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:33.674 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.672+0000 7f6e91863640 1 -- 172.21.15.156:0/1545008282 shutdown_connections 2024-03-25T18:10:33.674 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.672+0000 7f6e91863640 1 -- 172.21.15.156:0/1545008282 >> 172.21.15.156:0/1545008282 conn(0x7f6e8c0b4a80 msgr2=0x7f6e8c0afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:33.674 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.672+0000 7f6e91863640 1 -- 172.21.15.156:0/1545008282 shutdown_connections 2024-03-25T18:10:33.674 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.672+0000 7f6e91863640 1 -- 172.21.15.156:0/1545008282 wait complete. 2024-03-25T18:10:33.674 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:33.672+0000 7f6e91863640 1 librados: shutdown 2024-03-25T18:10:33.686 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:38.513 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:02:44.504 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:38.563 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:02:44.529 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:38.566 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:02:44.533 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:38.575 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:02:44.566 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:38.577 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:02:44.544 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:38.606 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:02:44.596 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:38.643 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:02:44.633 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:38.649 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:02:44.640 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:38.687 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:02:44.677 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:40.493 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:40.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.605+0000 7f555e2c1640 10 monclient: get_monmap_and_config 2024-03-25T18:10:40.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.605+0000 7f555e2c1640 10 monclient: build_initial_monmap 2024-03-25T18:10:40.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.605+0000 7f555e2c1640 10 monclient: monmap: 2024-03-25T18:10:40.607 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:10:40.607 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:40.607 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:10:40.606783+0000 2024-03-25T18:10:40.607 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:10:40.606783+0000 2024-03-25T18:10:40.607 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:40.607 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:40.607 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:10:40.607 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:10:40.607 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:10:40.608 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:40.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.606+0000 7f555e2c1640 1 Processor -- start 2024-03-25T18:10:40.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.606+0000 7f555e2c1640 1 -- start start 2024-03-25T18:10:40.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.606+0000 7f555e2c1640 10 monclient: init 2024-03-25T18:10:40.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.607+0000 7f555e2c1640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:40.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.607+0000 7f555e2c1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:10:40.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.607+0000 7f555e2c1640 10 monclient(hunting): picked mon.noname-a con 0x7f55580a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:10:40.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.607+0000 7f555e2c1640 10 monclient(hunting): picked mon.noname-b con 0x7f555813f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:10:40.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.607+0000 7f555e2c1640 10 monclient(hunting): picked mon.noname-c con 0x7f55581430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:10:40.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.607+0000 7f555e2c1640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5558142520 con 0x7f55580a98f0 2024-03-25T18:10:40.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.607+0000 7f555e2c1640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55580b28a0 con 0x7f55581430c0 2024-03-25T18:10:40.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.607+0000 7f555e2c1640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55580b2a30 con 0x7f555813f710 2024-03-25T18:10:40.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.607+0000 7f555e2c1640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:40.610 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.607+0000 7f555e2c1640 10 monclient(hunting): authenticate will time out at 7121.918447s 2024-03-25T18:10:40.614 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.613+0000 7f5557fff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f55580a98f0 0x7f55580a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:44112/0 (socket says 172.21.15.156:44112) 2024-03-25T18:10:40.614 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.613+0000 7f5557fff640 1 -- 172.21.15.156:0/2483033322 learned_addr learned my addr 172.21.15.156:0/2483033322 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:10:40.624 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.624+0000 7f5556ffd640 1 -- 172.21.15.156:0/2483033322 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902472245 0 0) 0x7f55580b28a0 con 0x7f55581430c0 2024-03-25T18:10:40.624 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.624+0000 7f5556ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:40.624 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.624+0000 7f5556ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:40.625 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.624+0000 7f5556ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:40.625 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.624+0000 7f5556ffd640 10 monclient(hunting): my global_id is 4324 2024-03-25T18:10:40.625 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.624+0000 7f5556ffd640 1 -- 172.21.15.156:0/2483033322 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5540003520 con 0x7f55581430c0 2024-03-25T18:10:40.627 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.625+0000 7f5556ffd640 1 -- 172.21.15.156:0/2483033322 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3009084919 0 0) 0x7f55580b2a30 con 0x7f555813f710 2024-03-25T18:10:40.627 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.625+0000 7f5556ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:40.627 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.625+0000 7f5556ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:40.627 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.625+0000 7f5556ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:40.627 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.625+0000 7f5556ffd640 10 monclient(hunting): my global_id is 4337 2024-03-25T18:10:40.627 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.625+0000 7f5556ffd640 1 -- 172.21.15.156:0/2483033322 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55580b28a0 con 0x7f555813f710 2024-03-25T18:10:40.627 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.626+0000 7f5556ffd640 1 -- 172.21.15.156:0/2483033322 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 529283883 0 0) 0x7f5558142520 con 0x7f55580a98f0 2024-03-25T18:10:40.627 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.626+0000 7f5556ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:40.627 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.626+0000 7f5556ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:40.628 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.626+0000 7f5556ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:40.628 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.626+0000 7f5556ffd640 10 monclient(hunting): my global_id is 4329 2024-03-25T18:10:40.628 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.626+0000 7f5556ffd640 1 -- 172.21.15.156:0/2483033322 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55580b2a30 con 0x7f55580a98f0 2024-03-25T18:10:40.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.630+0000 7f5556ffd640 1 -- 172.21.15.156:0/2483033322 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2735836477 0 0) 0x7f5540003520 con 0x7f55581430c0 2024-03-25T18:10:40.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.630+0000 7f5556ffd640 1 -- 172.21.15.156:0/2483033322 >> v1:172.21.15.156:6790/0 conn(0x7f555813f710 legacy=0x7f55581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:40.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.630+0000 7f5556ffd640 1 -- 172.21.15.156:0/2483033322 >> v1:172.21.15.156:6789/0 conn(0x7f55580a98f0 legacy=0x7f55580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:40.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.630+0000 7f5556ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:10:40.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.630+0000 7f5556ffd640 1 monclient: found mon.noname-c 2024-03-25T18:10:40.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.630+0000 7f5556ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:40.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.630+0000 7f5556ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:10:40.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.630+0000 7f5556ffd640 1 -- 172.21.15.156:0/2483033322 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55580b44c0 con 0x7f55581430c0 2024-03-25T18:10:40.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.631+0000 7f5556ffd640 10 monclient: _finish_auth 0 2024-03-25T18:10:40.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.631+0000 7f5556ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:10:40.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.631+0000 7f5556ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:40.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.631+0000 7f5556ffd640 1 -- 172.21.15.156:0/2483033322 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5550001a10 con 0x7f55581430c0 2024-03-25T18:10:40.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.631+0000 7f5556ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:40.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.631+0000 7f5556ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:40.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.631+0000 7f5556ffd640 10 monclient: dump: 2024-03-25T18:10:40.632 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:40.633 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:40.633 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:40.633 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:40.633 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:40.633 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:40.633 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:40.633 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:40.633 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:40.633 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:40.633 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.631+0000 7f555e2c1640 5 monclient: authenticate success, global_id 4324 2024-03-25T18:10:40.633 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.631+0000 7f555e2c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:40.642 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.642+0000 7f5556ffd640 1 -- 172.21.15.156:0/2483033322 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5550001b70 con 0x7f55581430c0 2024-03-25T18:10:40.642 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.642+0000 7f5556ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:40.642 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.642+0000 7f5556ffd640 1 -- 172.21.15.156:0/2483033322 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5550001f80 con 0x7f55581430c0 2024-03-25T18:10:40.642 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.642+0000 7f5556ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:40.643 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.642+0000 7f5556ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:40.643 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.642+0000 7f5556ffd640 10 monclient: dump: 2024-03-25T18:10:40.643 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:40.643 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:40.643 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:40.643 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:40.643 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:40.643 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:40.643 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:40.643 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:40.643 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:40.643 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:40.643 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.642+0000 7f555e2c1640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:40.643 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.642+0000 7f555e2c1640 4 set_mon_vals no callback set 2024-03-25T18:10:40.645 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.643+0000 7f555e2c1640 10 monclient: shutdown 2024-03-25T18:10:40.645 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.643+0000 7f555e2c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:40.645 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.643+0000 7f555e2c1640 1 -- 172.21.15.156:0/2483033322 >> v1:172.21.15.194:6789/0 conn(0x7f55581430c0 legacy=0x7f555814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:40.645 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.643+0000 7f555e2c1640 1 -- 172.21.15.156:0/2483033322 shutdown_connections 2024-03-25T18:10:40.645 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.643+0000 7f555e2c1640 1 -- 172.21.15.156:0/2483033322 >> 172.21.15.156:0/2483033322 conn(0x7f55580b7ea0 msgr2=0x7f55580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:40.645 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.643+0000 7f555e2c1640 1 -- 172.21.15.156:0/2483033322 shutdown_connections 2024-03-25T18:10:40.645 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.643+0000 7f555e2c1640 1 -- 172.21.15.156:0/2483033322 wait complete. 2024-03-25T18:10:40.645 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.643+0000 7f555e2c1640 10 monclient: build_initial_monmap 2024-03-25T18:10:40.645 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.643+0000 7f555e2c1640 10 monclient: monmap: 2024-03-25T18:10:40.645 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:10:40.645 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:40.645 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:10:40.645 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:10:40.645 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:40.645 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:10:40.645 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:10:40.646 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:10:40.646 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:10:40.646 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:40.646 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.644+0000 7f555e2c1640 1 librados: starting msgr at 2024-03-25T18:10:40.646 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.644+0000 7f555e2c1640 1 librados: starting objecter 2024-03-25T18:10:40.646 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.644+0000 7f555e2c1640 1 Processor -- start 2024-03-25T18:10:40.647 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.644+0000 7f555e2c1640 1 -- start start 2024-03-25T18:10:40.647 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.644+0000 7f555e2c1640 1 librados: setting wanted keys 2024-03-25T18:10:40.647 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.644+0000 7f555e2c1640 1 librados: calling monclient init 2024-03-25T18:10:40.647 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.644+0000 7f555e2c1640 10 monclient: init 2024-03-25T18:10:40.647 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.645+0000 7f555e2c1640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:40.647 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.645+0000 7f555e2c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:10:40.647 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.645+0000 7f555e2c1640 10 monclient(hunting): picked mon.noname-b con 0x7f55580a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:10:40.647 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.645+0000 7f555e2c1640 10 monclient(hunting): picked mon.noname-c con 0x7f555813f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:10:40.647 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.645+0000 7f555e2c1640 10 monclient(hunting): picked mon.noname-a con 0x7f55581430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:10:40.647 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.645+0000 7f555e2c1640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55581daca0 con 0x7f55581430c0 2024-03-25T18:10:40.647 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.645+0000 7f555e2c1640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5558138ab0 con 0x7f55580a98f0 2024-03-25T18:10:40.648 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.645+0000 7f555e2c1640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55580aba90 con 0x7f555813f710 2024-03-25T18:10:40.648 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.645+0000 7f555e2c1640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:40.648 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.645+0000 7f555e2c1640 10 monclient(hunting): authenticate will time out at 7121.956443s 2024-03-25T18:10:40.662 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.661+0000 7f55577fe640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f555813f710 0x7f55581db130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:57536/0 (socket says 172.21.15.156:57536) 2024-03-25T18:10:40.662 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.661+0000 7f55577fe640 1 -- 172.21.15.156:0/2507611929 learned_addr learned my addr 172.21.15.156:0/2507611929 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:10:40.664 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.663+0000 7f5554ff9640 10 client.?.objecter ms_handle_connect 0x7f555813f710 2024-03-25T18:10:40.664 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.663+0000 7f5554ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:40.666 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.665+0000 7f5554ff9640 10 client.?.objecter ms_handle_connect 0x7f55580a98f0 2024-03-25T18:10:40.666 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.665+0000 7f5554ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:40.676 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.675+0000 7f5554ff9640 1 -- 172.21.15.156:0/2507611929 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3976324188 0 0) 0x7f55580aba90 con 0x7f555813f710 2024-03-25T18:10:40.676 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.675+0000 7f5554ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:10:40.676 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.675+0000 7f5554ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:40.676 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.675+0000 7f5554ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:40.676 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.675+0000 7f5554ff9640 10 monclient(hunting): my global_id is 4340 2024-03-25T18:10:40.676 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.675+0000 7f5554ff9640 1 -- 172.21.15.156:0/2507611929 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5544004eb0 con 0x7f555813f710 2024-03-25T18:10:40.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.676+0000 7f5554ff9640 10 client.?.objecter ms_handle_connect 0x7f55581430c0 2024-03-25T18:10:40.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.676+0000 7f5554ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:40.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.685+0000 7f5554ff9640 1 -- 172.21.15.156:0/2507611929 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125453374 0 0) 0x7f5558138ab0 con 0x7f55580a98f0 2024-03-25T18:10:40.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.685+0000 7f5554ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:10:40.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.685+0000 7f5554ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:40.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.685+0000 7f5554ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:40.687 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.685+0000 7f5554ff9640 10 monclient(hunting): my global_id is 4327 2024-03-25T18:10:40.687 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.685+0000 7f5554ff9640 1 -- 172.21.15.156:0/2507611929 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55580aba90 con 0x7f55580a98f0 2024-03-25T18:10:40.691 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.691+0000 7f5554ff9640 1 -- 172.21.15.156:0/2507611929 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102639456 0 0) 0x7f55581daca0 con 0x7f55581430c0 2024-03-25T18:10:40.692 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.691+0000 7f5554ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:10:40.692 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.691+0000 7f5554ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:40.692 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.691+0000 7f5554ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:40.692 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.691+0000 7f5554ff9640 10 monclient(hunting): my global_id is 4332 2024-03-25T18:10:40.692 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.691+0000 7f5554ff9640 1 -- 172.21.15.156:0/2507611929 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5558138ab0 con 0x7f55581430c0 2024-03-25T18:10:40.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.695+0000 7f5554ff9640 1 -- 172.21.15.156:0/2507611929 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2664253392 0 0) 0x7f5544004eb0 con 0x7f555813f710 2024-03-25T18:10:40.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.695+0000 7f5554ff9640 1 -- 172.21.15.156:0/2507611929 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55581daca0 con 0x7f555813f710 2024-03-25T18:10:40.696 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.695+0000 7f5554ff9640 1 -- 172.21.15.156:0/2507611929 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f554c001a30 con 0x7f555813f710 2024-03-25T18:10:40.696 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.695+0000 7f5554ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:40.696 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.695+0000 7f5554ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:40.696 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.695+0000 7f5554ff9640 10 monclient(hunting): dump: 2024-03-25T18:10:40.696 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:40.696 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:40.696 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:40.696 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:40.696 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:40.696 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:40.696 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:40.696 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:40.696 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:40.696 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:40.696 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.695+0000 7f5554ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:40.705 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.704+0000 7f5554ff9640 1 -- 172.21.15.156:0/2507611929 <== mon.2 v1:172.21.15.156:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 960315190 0 0) 0x7f55581daca0 con 0x7f555813f710 2024-03-25T18:10:40.705 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.704+0000 7f5554ff9640 1 -- 172.21.15.156:0/2507611929 >> v1:172.21.15.194:6789/0 conn(0x7f55580a98f0 legacy=0x7f55581383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:40.705 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.704+0000 7f5554ff9640 1 -- 172.21.15.156:0/2507611929 >> v1:172.21.15.156:6789/0 conn(0x7f55581430c0 legacy=0x7f55581d47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:40.705 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.704+0000 7f5554ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:10:40.706 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.704+0000 7f5554ff9640 1 monclient: found mon.c 2024-03-25T18:10:40.706 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.704+0000 7f5554ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:40.706 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.704+0000 7f5554ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:10:40.706 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.704+0000 7f5554ff9640 1 -- 172.21.15.156:0/2507611929 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55581d41e0 con 0x7f555813f710 2024-03-25T18:10:40.706 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.704+0000 7f5554ff9640 10 monclient: _finish_auth 0 2024-03-25T18:10:40.706 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.704+0000 7f5554ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:10:40.706 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.704+0000 7f5554ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:40.706 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.704+0000 7f555e2c1640 5 monclient: authenticate success, global_id 4340 2024-03-25T18:10:40.706 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.704+0000 7f555e2c1640 10 monclient: _renew_subs 2024-03-25T18:10:40.706 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.704+0000 7f555e2c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:10:40.706 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.704+0000 7f555e2c1640 1 -- 172.21.15.156:0/2507611929 --> v1:172.21.15.156:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55581db840 con 0x7f555813f710 2024-03-25T18:10:40.706 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.704+0000 7f555e2c1640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:40.706 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.704+0000 7f555e2c1640 10 monclient: _renew_subs 2024-03-25T18:10:40.706 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.704+0000 7f555e2c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:10:40.706 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.704+0000 7f555e2c1640 1 -- 172.21.15.156:0/2507611929 --> v1:172.21.15.156:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5558138f00 con 0x7f555813f710 2024-03-25T18:10:40.707 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.704+0000 7f555e2c1640 1 librados: init done 2024-03-25T18:10:40.707 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.705+0000 7f555e2c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:40.707 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.705+0000 7f555e2c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:40.707 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.705+0000 7f555e2c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:10:40.707 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:40.705+0000 7f555e2c1640 1 -- 172.21.15.156:0/2507611929 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55581dae50 con 0x7f555813f710 2024-03-25T18:10:41.020 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.019+0000 7f5554ff9640 1 -- 172.21.15.156:0/2507611929 <== mon.2 v1:172.21.15.156:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f554c001c70 con 0x7f555813f710 2024-03-25T18:10:41.021 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.019+0000 7f5554ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:41.021 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.019+0000 7f5554ff9640 1 -- 172.21.15.156:0/2507611929 <== mon.2 v1:172.21.15.156:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f554c0038c0 con 0x7f555813f710 2024-03-25T18:10:41.021 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.019+0000 7f5554ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:41.021 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.019+0000 7f5554ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:41.021 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.019+0000 7f5554ff9640 10 monclient: dump: 2024-03-25T18:10:41.021 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:41.021 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:41.021 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:41.021 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:41.021 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:41.021 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:41.021 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:41.021 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:41.022 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:41.022 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:41.022 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.019+0000 7f55567fc640 4 set_mon_vals no callback set 2024-03-25T18:10:41.022 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.020+0000 7f5554ff9640 1 -- 172.21.15.156:0/2507611929 <== mon.2 v1:172.21.15.156:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f554c003aa0 con 0x7f555813f710 2024-03-25T18:10:41.022 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.021+0000 7f5557fff640 10 monclient: get_auth_request con 0x7f554406ddf0 auth_method 0 2024-03-25T18:10:41.022 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.021+0000 7f5554ff9640 10 client.4340.objecter ms_handle_connect 0x7f554406ddf0 2024-03-25T18:10:41.029 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.028+0000 7f5554ff9640 1 -- 172.21.15.156:0/2507611929 <== mon.2 v1:172.21.15.156:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f554c054710 con 0x7f555813f710 2024-03-25T18:10:41.029 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.028+0000 7f5554ff9640 10 client.4340.objecter ms_dispatch 0x7f5558000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:41.029 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.028+0000 7f5554ff9640 3 client.4340.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:41.029 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.028+0000 7f5554ff9640 3 client.4340.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:41.029 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.028+0000 7f5554ff9640 20 client.4340.objecter dump_active .. 0 homeless 2024-03-25T18:10:41.030 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.028+0000 7f5554ff9640 1 -- 172.21.15.156:0/2507611929 <== mon.2 v1:172.21.15.156:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f554c050220 con 0x7f555813f710 2024-03-25T18:10:41.030 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.028+0000 7f5554ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:41.030 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.028+0000 7f5554ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:41.269 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.268+0000 7f555e2c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:41.269 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.268+0000 7f555e2c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:41.270 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.268+0000 7f555e2c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:10:41.270 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.268+0000 7f555e2c1640 1 -- 172.21.15.156:0/2507611929 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55580627b0 con 0x7f555813f710 2024-03-25T18:10:41.308 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.307+0000 7f5554ff9640 1 -- 172.21.15.156:0/2507611929 <== mon.2 v1:172.21.15.156:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f554c052150 con 0x7f555813f710 2024-03-25T18:10:41.308 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.307+0000 7f5554ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:41.308 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.307+0000 7f5554ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:41.308 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:10:41.308 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:41.310 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.309+0000 7f555e2c1640 10 librados: watch_flush enter 2024-03-25T18:10:41.311 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.309+0000 7f555e2c1640 10 librados: watch_flush exit 2024-03-25T18:10:41.311 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.309+0000 7f555e2c1640 20 client.4340.objecter shutdown clearing up homeless session... 2024-03-25T18:10:41.311 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.309+0000 7f555e2c1640 10 client.4340.objecter successfully canceled tick 2024-03-25T18:10:41.311 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.309+0000 7f555e2c1640 1 -- 172.21.15.156:0/2507611929 >> v1:172.21.15.194:6800/2946427713 conn(0x7f554406ddf0 legacy=0x7f55440702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:41.311 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.309+0000 7f555e2c1640 10 monclient: shutdown 2024-03-25T18:10:41.311 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.309+0000 7f555e2c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:41.311 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.309+0000 7f555e2c1640 1 -- 172.21.15.156:0/2507611929 >> v1:172.21.15.156:6790/0 conn(0x7f555813f710 legacy=0x7f55581db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:41.311 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.309+0000 7f555e2c1640 1 -- 172.21.15.156:0/2507611929 shutdown_connections 2024-03-25T18:10:41.311 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.309+0000 7f555e2c1640 1 -- 172.21.15.156:0/2507611929 >> 172.21.15.156:0/2507611929 conn(0x7f55580b7ea0 msgr2=0x7f5558074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:41.311 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.310+0000 7f555e2c1640 1 -- 172.21.15.156:0/2507611929 shutdown_connections 2024-03-25T18:10:41.311 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.310+0000 7f555e2c1640 1 -- 172.21.15.156:0/2507611929 wait complete. 2024-03-25T18:10:41.311 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:41.310+0000 7f555e2c1640 1 librados: shutdown 2024-03-25T18:10:41.323 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:48.130 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:48.239 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.239+0000 7f2e3e9c8640 10 monclient: get_monmap_and_config 2024-03-25T18:10:48.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.239+0000 7f2e3e9c8640 10 monclient: build_initial_monmap 2024-03-25T18:10:48.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.239+0000 7f2e3e9c8640 10 monclient: monmap: 2024-03-25T18:10:48.240 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:10:48.240 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:48.240 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:10:48.240167+0000 2024-03-25T18:10:48.240 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:10:48.240167+0000 2024-03-25T18:10:48.240 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:48.240 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:48.240 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:10:48.240 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:10:48.240 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:10:48.240 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:48.242 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.239+0000 7f2e3e9c8640 1 Processor -- start 2024-03-25T18:10:48.242 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.239+0000 7f2e3e9c8640 1 -- start start 2024-03-25T18:10:48.242 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.239+0000 7f2e3e9c8640 10 monclient: init 2024-03-25T18:10:48.242 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.240+0000 7f2e3e9c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:48.242 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.240+0000 7f2e3e9c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:10:48.242 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.240+0000 7f2e3e9c8640 10 monclient(hunting): picked mon.noname-c con 0x7f2e380b4d60 addr v1:172.21.15.194:6789/0 2024-03-25T18:10:48.242 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.240+0000 7f2e3e9c8640 10 monclient(hunting): picked mon.noname-a con 0x7f2e380b1730 addr v1:172.21.15.156:6789/0 2024-03-25T18:10:48.242 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.240+0000 7f2e3e9c8640 10 monclient(hunting): picked mon.noname-b con 0x7f2e3814be90 addr v1:172.21.15.156:6790/0 2024-03-25T18:10:48.242 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.240+0000 7f2e3e9c8640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e380af940 con 0x7f2e380b1730 2024-03-25T18:10:48.242 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.240+0000 7f2e3e9c8640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e380b0a70 con 0x7f2e380b4d60 2024-03-25T18:10:48.242 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.240+0000 7f2e3e9c8640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e380b0d50 con 0x7f2e3814be90 2024-03-25T18:10:48.242 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.240+0000 7f2e3e9c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:48.242 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.240+0000 7f2e3e9c8640 10 monclient(hunting): authenticate will time out at 7129.551326s 2024-03-25T18:10:48.243 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.241+0000 7f2e3cf65640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f2e3814be90 0x7f2e3814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:43292/0 (socket says 172.21.15.156:43292) 2024-03-25T18:10:48.243 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.241+0000 7f2e3cf65640 1 -- 172.21.15.156:0/2944033576 learned_addr learned my addr 172.21.15.156:0/2944033576 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:10:48.302 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.300+0000 7f2e36ffd640 1 -- 172.21.15.156:0/2944033576 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 149211966 0 0) 0x7f2e380af940 con 0x7f2e380b1730 2024-03-25T18:10:48.302 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.301+0000 7f2e36ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:48.302 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.301+0000 7f2e36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:48.302 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.301+0000 7f2e36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:48.302 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.301+0000 7f2e36ffd640 10 monclient(hunting): my global_id is 4335 2024-03-25T18:10:48.302 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.301+0000 7f2e36ffd640 1 -- 172.21.15.156:0/2944033576 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e24003520 con 0x7f2e380b1730 2024-03-25T18:10:48.315 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.314+0000 7f2e36ffd640 1 -- 172.21.15.156:0/2944033576 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 593422733 0 0) 0x7f2e24003520 con 0x7f2e380b1730 2024-03-25T18:10:48.315 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.314+0000 7f2e36ffd640 1 -- 172.21.15.156:0/2944033576 >> v1:172.21.15.156:6790/0 conn(0x7f2e3814be90 legacy=0x7f2e3814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:48.316 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.314+0000 7f2e36ffd640 1 -- 172.21.15.156:0/2944033576 >> v1:172.21.15.194:6789/0 conn(0x7f2e380b4d60 legacy=0x7f2e3813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:48.316 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.314+0000 7f2e36ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:10:48.316 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.314+0000 7f2e36ffd640 1 monclient: found mon.noname-a 2024-03-25T18:10:48.316 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.314+0000 7f2e36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:48.316 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.314+0000 7f2e36ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:10:48.316 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.314+0000 7f2e36ffd640 1 -- 172.21.15.156:0/2944033576 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e380b1030 con 0x7f2e380b1730 2024-03-25T18:10:48.316 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.314+0000 7f2e36ffd640 10 monclient: _finish_auth 0 2024-03-25T18:10:48.316 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.314+0000 7f2e36ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:10:48.316 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.314+0000 7f2e36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:48.316 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.315+0000 7f2e3e9c8640 5 monclient: authenticate success, global_id 4335 2024-03-25T18:10:48.316 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.314+0000 7f2e36ffd640 1 -- 172.21.15.156:0/2944033576 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f2e20001690 con 0x7f2e380b1730 2024-03-25T18:10:48.316 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.315+0000 7f2e3e9c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:48.316 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.315+0000 7f2e36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:48.317 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.315+0000 7f2e36ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:48.317 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.315+0000 7f2e36ffd640 10 monclient: dump: 2024-03-25T18:10:48.317 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:48.317 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:48.317 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:48.317 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:48.317 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:48.317 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:48.317 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:48.317 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:48.317 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:48.317 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:48.317 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.315+0000 7f2e3e9c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:48.334 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.333+0000 7f2e36ffd640 1 -- 172.21.15.156:0/2944033576 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e200017f0 con 0x7f2e380b1730 2024-03-25T18:10:48.334 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.333+0000 7f2e36ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:48.335 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.333+0000 7f2e36ffd640 1 -- 172.21.15.156:0/2944033576 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f2e20001c00 con 0x7f2e380b1730 2024-03-25T18:10:48.335 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.333+0000 7f2e3e9c8640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:48.335 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.333+0000 7f2e3e9c8640 4 set_mon_vals no callback set 2024-03-25T18:10:48.335 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.334+0000 7f2e36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:48.335 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.334+0000 7f2e36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:48.335 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.334+0000 7f2e36ffd640 10 monclient: dump: 2024-03-25T18:10:48.335 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:48.335 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:48.335 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:48.335 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:48.335 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:48.335 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:48.335 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:48.335 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:48.336 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:48.336 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:48.337 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.334+0000 7f2e3e9c8640 10 monclient: shutdown 2024-03-25T18:10:48.337 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.334+0000 7f2e3e9c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:48.337 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.334+0000 7f2e3e9c8640 1 -- 172.21.15.156:0/2944033576 >> v1:172.21.15.156:6789/0 conn(0x7f2e380b1730 legacy=0x7f2e380b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:48.337 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.335+0000 7f2e3e9c8640 1 -- 172.21.15.156:0/2944033576 shutdown_connections 2024-03-25T18:10:48.337 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.335+0000 7f2e3e9c8640 1 -- 172.21.15.156:0/2944033576 >> 172.21.15.156:0/2944033576 conn(0x7f2e380617a0 msgr2=0x7f2e38061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:48.337 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.335+0000 7f2e3e9c8640 1 -- 172.21.15.156:0/2944033576 shutdown_connections 2024-03-25T18:10:48.337 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.335+0000 7f2e3e9c8640 1 -- 172.21.15.156:0/2944033576 wait complete. 2024-03-25T18:10:48.337 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.336+0000 7f2e3e9c8640 10 monclient: build_initial_monmap 2024-03-25T18:10:48.337 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.336+0000 7f2e3e9c8640 10 monclient: monmap: 2024-03-25T18:10:48.338 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:10:48.338 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:48.338 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:10:48.338 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:10:48.338 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:48.338 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:10:48.338 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:10:48.338 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:10:48.338 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:10:48.338 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:48.338 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.336+0000 7f2e3e9c8640 1 librados: starting msgr at 2024-03-25T18:10:48.338 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.336+0000 7f2e3e9c8640 1 librados: starting objecter 2024-03-25T18:10:48.338 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.336+0000 7f2e3e9c8640 1 Processor -- start 2024-03-25T18:10:48.338 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.336+0000 7f2e3e9c8640 1 -- start start 2024-03-25T18:10:48.339 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.336+0000 7f2e3e9c8640 1 librados: setting wanted keys 2024-03-25T18:10:48.339 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.336+0000 7f2e3e9c8640 1 librados: calling monclient init 2024-03-25T18:10:48.339 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.336+0000 7f2e3e9c8640 10 monclient: init 2024-03-25T18:10:48.340 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.337+0000 7f2e3e9c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:48.340 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.337+0000 7f2e3e9c8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:10:48.340 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.337+0000 7f2e3e9c8640 10 monclient(hunting): picked mon.noname-a con 0x7f2e380b1730 addr v1:172.21.15.156:6789/0 2024-03-25T18:10:48.340 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.337+0000 7f2e3e9c8640 10 monclient(hunting): picked mon.noname-b con 0x7f2e380b4d60 addr v1:172.21.15.194:6789/0 2024-03-25T18:10:48.340 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.337+0000 7f2e3e9c8640 10 monclient(hunting): picked mon.noname-c con 0x7f2e3814be90 addr v1:172.21.15.156:6790/0 2024-03-25T18:10:48.340 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.337+0000 7f2e3e9c8640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e381420d0 con 0x7f2e380b1730 2024-03-25T18:10:48.340 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.337+0000 7f2e3e9c8640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e38138270 con 0x7f2e380b4d60 2024-03-25T18:10:48.340 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.337+0000 7f2e3e9c8640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e38138400 con 0x7f2e3814be90 2024-03-25T18:10:48.340 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.337+0000 7f2e3e9c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:48.340 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.337+0000 7f2e3e9c8640 10 monclient(hunting): authenticate will time out at 7129.648462s 2024-03-25T18:10:48.350 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.349+0000 7f2e377fe640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f2e380b4d60 0x7f2e381405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:40020/0 (socket says 172.21.15.156:40020) 2024-03-25T18:10:48.350 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.349+0000 7f2e377fe640 1 -- 172.21.15.156:0/1030061750 learned_addr learned my addr 172.21.15.156:0/1030061750 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:10:48.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.350+0000 7f2e34ff9640 10 client.?.objecter ms_handle_connect 0x7f2e380b4d60 2024-03-25T18:10:48.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.350+0000 7f2e34ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:48.359 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.359+0000 7f2e34ff9640 10 client.?.objecter ms_handle_connect 0x7f2e380b1730 2024-03-25T18:10:48.359 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.359+0000 7f2e34ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:48.373 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.372+0000 7f2e34ff9640 10 client.?.objecter ms_handle_connect 0x7f2e3814be90 2024-03-25T18:10:48.373 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.372+0000 7f2e34ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:48.380 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.379+0000 7f2e34ff9640 1 -- 172.21.15.156:0/1030061750 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308725631 0 0) 0x7f2e381420d0 con 0x7f2e380b1730 2024-03-25T18:10:48.380 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.379+0000 7f2e34ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:10:48.380 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.379+0000 7f2e34ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:48.380 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.379+0000 7f2e34ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:48.380 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.379+0000 7f2e34ff9640 10 monclient(hunting): my global_id is 4338 2024-03-25T18:10:48.380 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.379+0000 7f2e34ff9640 1 -- 172.21.15.156:0/1030061750 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e18004eb0 con 0x7f2e380b1730 2024-03-25T18:10:48.438 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.437+0000 7f2e34ff9640 1 -- 172.21.15.156:0/1030061750 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1928283703 0 0) 0x7f2e38138270 con 0x7f2e380b4d60 2024-03-25T18:10:48.438 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.437+0000 7f2e34ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:10:48.438 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.437+0000 7f2e34ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:48.438 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.437+0000 7f2e34ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:48.438 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.437+0000 7f2e34ff9640 10 monclient(hunting): my global_id is 4333 2024-03-25T18:10:48.438 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.437+0000 7f2e34ff9640 1 -- 172.21.15.156:0/1030061750 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e381420d0 con 0x7f2e380b4d60 2024-03-25T18:10:48.441 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.440+0000 7f2e34ff9640 1 -- 172.21.15.156:0/1030061750 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473547305 0 0) 0x7f2e38138400 con 0x7f2e3814be90 2024-03-25T18:10:48.441 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.440+0000 7f2e34ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:10:48.441 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.440+0000 7f2e34ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:48.441 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.440+0000 7f2e34ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:48.441 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.440+0000 7f2e34ff9640 10 monclient(hunting): my global_id is 4343 2024-03-25T18:10:48.441 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.440+0000 7f2e34ff9640 1 -- 172.21.15.156:0/1030061750 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e38138270 con 0x7f2e3814be90 2024-03-25T18:10:48.457 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.456+0000 7f2e34ff9640 1 -- 172.21.15.156:0/1030061750 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 412016499 0 0) 0x7f2e381420d0 con 0x7f2e380b4d60 2024-03-25T18:10:48.457 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.456+0000 7f2e34ff9640 1 -- 172.21.15.156:0/1030061750 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e38138400 con 0x7f2e380b4d60 2024-03-25T18:10:48.457 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.456+0000 7f2e34ff9640 1 -- 172.21.15.156:0/1030061750 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f2e20001690 con 0x7f2e380b4d60 2024-03-25T18:10:48.457 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.456+0000 7f2e34ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:48.457 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.456+0000 7f2e34ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:48.457 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.456+0000 7f2e34ff9640 10 monclient(hunting): dump: 2024-03-25T18:10:48.457 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:48.457 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:48.457 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:48.457 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:48.457 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:48.458 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:48.458 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:48.458 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:48.458 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:48.458 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:48.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.456+0000 7f2e34ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:48.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.459+0000 7f2e34ff9640 1 -- 172.21.15.156:0/1030061750 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3814534356 0 0) 0x7f2e18004eb0 con 0x7f2e380b1730 2024-03-25T18:10:48.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.459+0000 7f2e34ff9640 1 -- 172.21.15.156:0/1030061750 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e381420d0 con 0x7f2e380b1730 2024-03-25T18:10:48.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.459+0000 7f2e34ff9640 1 -- 172.21.15.156:0/1030061750 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f2e280018c0 con 0x7f2e380b1730 2024-03-25T18:10:48.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.459+0000 7f2e34ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:48.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.459+0000 7f2e34ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:48.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.459+0000 7f2e34ff9640 10 monclient(hunting): dump: 2024-03-25T18:10:48.460 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:48.460 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:48.460 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:48.460 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:48.461 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:48.461 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:48.461 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:48.461 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:48.461 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:48.461 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:48.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.468+0000 7f2e34ff9640 1 -- 172.21.15.156:0/1030061750 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1016455818 0 0) 0x7f2e38138270 con 0x7f2e3814be90 2024-03-25T18:10:48.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.468+0000 7f2e34ff9640 1 -- 172.21.15.156:0/1030061750 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e18004eb0 con 0x7f2e3814be90 2024-03-25T18:10:48.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.468+0000 7f2e34ff9640 1 -- 172.21.15.156:0/1030061750 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f2e30001a60 con 0x7f2e3814be90 2024-03-25T18:10:48.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.468+0000 7f2e34ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:48.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.468+0000 7f2e34ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:48.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.468+0000 7f2e34ff9640 10 monclient(hunting): dump: 2024-03-25T18:10:48.469 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:48.469 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:48.469 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:48.469 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:48.469 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:48.469 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:48.469 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:48.469 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:48.470 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:48.470 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:48.474 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.472+0000 7f2e34ff9640 1 -- 172.21.15.156:0/1030061750 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 794088746 0 0) 0x7f2e38138400 con 0x7f2e380b4d60 2024-03-25T18:10:48.474 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.472+0000 7f2e34ff9640 1 -- 172.21.15.156:0/1030061750 >> v1:172.21.15.156:6790/0 conn(0x7f2e3814be90 legacy=0x7f2e38134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:48.474 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.472+0000 7f2e34ff9640 1 -- 172.21.15.156:0/1030061750 >> v1:172.21.15.156:6789/0 conn(0x7f2e380b1730 legacy=0x7f2e381db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:48.474 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.472+0000 7f2e34ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:10:48.474 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.472+0000 7f2e34ff9640 1 monclient: found mon.b 2024-03-25T18:10:48.474 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.472+0000 7f2e34ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:48.474 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.472+0000 7f2e34ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:10:48.474 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.472+0000 7f2e34ff9640 1 -- 172.21.15.156:0/1030061750 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e38142480 con 0x7f2e380b4d60 2024-03-25T18:10:48.474 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.472+0000 7f2e34ff9640 10 monclient: _finish_auth 0 2024-03-25T18:10:48.474 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.472+0000 7f2e34ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:10:48.474 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.472+0000 7f2e34ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:48.474 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.472+0000 7f2e3e9c8640 5 monclient: authenticate success, global_id 4333 2024-03-25T18:10:48.474 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.472+0000 7f2e3e9c8640 10 monclient: _renew_subs 2024-03-25T18:10:48.474 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.472+0000 7f2e3e9c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:10:48.474 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.472+0000 7f2e3e9c8640 1 -- 172.21.15.156:0/1030061750 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e381426d0 con 0x7f2e380b4d60 2024-03-25T18:10:48.475 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.473+0000 7f2e3e9c8640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:48.475 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.473+0000 7f2e3e9c8640 10 monclient: _renew_subs 2024-03-25T18:10:48.475 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.473+0000 7f2e3e9c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:10:48.475 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.473+0000 7f2e3e9c8640 1 -- 172.21.15.156:0/1030061750 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e381dba90 con 0x7f2e380b4d60 2024-03-25T18:10:48.475 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.473+0000 7f2e3e9c8640 1 librados: init done 2024-03-25T18:10:48.475 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.474+0000 7f2e3e9c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:48.476 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.474+0000 7f2e3e9c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:48.476 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.474+0000 7f2e3e9c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:10:48.476 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.474+0000 7f2e3e9c8640 1 -- 172.21.15.156:0/1030061750 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e381401f0 con 0x7f2e380b4d60 2024-03-25T18:10:48.748 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.747+0000 7f2e34ff9640 1 -- 172.21.15.156:0/1030061750 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e200038a0 con 0x7f2e380b4d60 2024-03-25T18:10:48.748 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.747+0000 7f2e34ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:48.748 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.747+0000 7f2e34ff9640 1 -- 172.21.15.156:0/1030061750 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f2e20003c70 con 0x7f2e380b4d60 2024-03-25T18:10:48.748 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.747+0000 7f2e34ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:48.748 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.747+0000 7f2e34ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:48.748 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.747+0000 7f2e34ff9640 10 monclient: dump: 2024-03-25T18:10:48.748 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:48.748 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:48.749 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:48.749 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:48.749 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:48.749 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:48.749 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:48.749 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:48.749 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:48.749 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:48.749 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.747+0000 7f2e367fc640 4 set_mon_vals no callback set 2024-03-25T18:10:48.750 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.748+0000 7f2e34ff9640 1 -- 172.21.15.156:0/1030061750 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f2e20003e90 con 0x7f2e380b4d60 2024-03-25T18:10:48.750 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.749+0000 7f2e37fff640 10 monclient: get_auth_request con 0x7f2e1806e0f0 auth_method 0 2024-03-25T18:10:48.750 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.749+0000 7f2e34ff9640 10 client.4333.objecter ms_handle_connect 0x7f2e1806e0f0 2024-03-25T18:10:48.754 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.753+0000 7f2e34ff9640 1 -- 172.21.15.156:0/1030061750 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f2e200548e0 con 0x7f2e380b4d60 2024-03-25T18:10:48.754 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.753+0000 7f2e34ff9640 10 client.4333.objecter ms_dispatch 0x7f2e38000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:48.754 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.753+0000 7f2e34ff9640 3 client.4333.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:48.754 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.753+0000 7f2e34ff9640 3 client.4333.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:48.754 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.753+0000 7f2e34ff9640 20 client.4333.objecter dump_active .. 0 homeless 2024-03-25T18:10:48.755 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.753+0000 7f2e34ff9640 1 -- 172.21.15.156:0/1030061750 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2e200503f0 con 0x7f2e380b4d60 2024-03-25T18:10:48.755 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.753+0000 7f2e34ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:48.755 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:48.753+0000 7f2e34ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:49.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:49.009+0000 7f2e3e9c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:49.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:49.009+0000 7f2e3e9c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:49.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:49.009+0000 7f2e3e9c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:10:49.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:49.009+0000 7f2e3e9c8640 1 -- 172.21.15.156:0/1030061750 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e380627b0 con 0x7f2e380b4d60 2024-03-25T18:10:49.039 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:49.038+0000 7f2e34ff9640 1 -- 172.21.15.156:0/1030061750 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f2e20052320 con 0x7f2e380b4d60 2024-03-25T18:10:49.039 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:49.038+0000 7f2e34ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:49.039 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:49.038+0000 7f2e34ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:49.039 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:10:49.039 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:49.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:49.041+0000 7f2e3e9c8640 10 librados: watch_flush enter 2024-03-25T18:10:49.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:49.041+0000 7f2e3e9c8640 10 librados: watch_flush exit 2024-03-25T18:10:49.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:49.041+0000 7f2e3e9c8640 20 client.4333.objecter shutdown clearing up homeless session... 2024-03-25T18:10:49.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:49.041+0000 7f2e3e9c8640 10 client.4333.objecter successfully canceled tick 2024-03-25T18:10:49.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:49.041+0000 7f2e3e9c8640 1 -- 172.21.15.156:0/1030061750 >> v1:172.21.15.194:6800/2946427713 conn(0x7f2e1806e0f0 legacy=0x7f2e180705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:49.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:49.041+0000 7f2e3e9c8640 10 monclient: shutdown 2024-03-25T18:10:49.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:49.041+0000 7f2e3e9c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:49.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:49.041+0000 7f2e3e9c8640 1 -- 172.21.15.156:0/1030061750 >> v1:172.21.15.194:6789/0 conn(0x7f2e380b4d60 legacy=0x7f2e381405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:49.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:49.041+0000 7f2e3e9c8640 1 -- 172.21.15.156:0/1030061750 shutdown_connections 2024-03-25T18:10:49.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:49.041+0000 7f2e3e9c8640 1 -- 172.21.15.156:0/1030061750 >> 172.21.15.156:0/1030061750 conn(0x7f2e380617a0 msgr2=0x7f2e380b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:49.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:49.041+0000 7f2e3e9c8640 1 -- 172.21.15.156:0/1030061750 shutdown_connections 2024-03-25T18:10:49.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:49.041+0000 7f2e3e9c8640 1 -- 172.21.15.156:0/1030061750 wait complete. 2024-03-25T18:10:49.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:49.041+0000 7f2e3e9c8640 1 librados: shutdown 2024-03-25T18:10:49.055 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:55.862 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:55.975 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:55.974+0000 7fd08ba25640 10 monclient: get_monmap_and_config 2024-03-25T18:10:55.975 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:55.974+0000 7fd08ba25640 10 monclient: build_initial_monmap 2024-03-25T18:10:55.975 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:55.974+0000 7fd08ba25640 10 monclient: monmap: 2024-03-25T18:10:55.975 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:10:55.975 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:55.975 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:10:55.975482+0000 2024-03-25T18:10:55.975 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:10:55.975482+0000 2024-03-25T18:10:55.975 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:55.976 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:55.976 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:10:55.976 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:10:55.976 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:10:55.976 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:55.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:55.975+0000 7fd08ba25640 1 Processor -- start 2024-03-25T18:10:55.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:55.975+0000 7fd08ba25640 1 -- start start 2024-03-25T18:10:55.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:55.975+0000 7fd08ba25640 10 monclient: init 2024-03-25T18:10:55.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:55.976+0000 7fd08ba25640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:55.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:55.976+0000 7fd08ba25640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:10:55.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:55.976+0000 7fd08ba25640 10 monclient(hunting): picked mon.noname-b con 0x7fd0840a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:10:55.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:55.976+0000 7fd08ba25640 10 monclient(hunting): picked mon.noname-a con 0x7fd08413f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:10:55.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:55.976+0000 7fd08ba25640 10 monclient(hunting): picked mon.noname-c con 0x7fd0841430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:10:55.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:55.976+0000 7fd08ba25640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd084142520 con 0x7fd08413f710 2024-03-25T18:10:55.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:55.976+0000 7fd08ba25640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0840b28a0 con 0x7fd0841430c0 2024-03-25T18:10:55.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:55.976+0000 7fd08ba25640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0840b2a30 con 0x7fd0840a98f0 2024-03-25T18:10:55.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:55.976+0000 7fd08ba25640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:55.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:55.976+0000 7fd08ba25640 10 monclient(hunting): authenticate will time out at 7137.286929s 2024-03-25T18:10:55.984 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:55.984+0000 7fd089fc2640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fd0841430c0 0x7fd08414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:49952/0 (socket says 172.21.15.156:49952) 2024-03-25T18:10:55.985 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:55.984+0000 7fd089fc2640 1 -- 172.21.15.156:0/4121091622 learned_addr learned my addr 172.21.15.156:0/4121091622 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:10:56.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:55.999+0000 7fd07bfff640 1 -- 172.21.15.156:0/4121091622 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521798711 0 0) 0x7fd0840b28a0 con 0x7fd0841430c0 2024-03-25T18:10:56.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:55.999+0000 7fd07bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:56.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:55.999+0000 7fd07bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:56.001 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:55.999+0000 7fd07bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:56.001 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:55.999+0000 7fd07bfff640 10 monclient(hunting): my global_id is 4336 2024-03-25T18:10:56.001 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:55.999+0000 7fd07bfff640 1 -- 172.21.15.156:0/4121091622 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd070003520 con 0x7fd0841430c0 2024-03-25T18:10:56.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.009+0000 7fd07bfff640 1 -- 172.21.15.156:0/4121091622 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341424285 0 0) 0x7fd084142520 con 0x7fd08413f710 2024-03-25T18:10:56.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.009+0000 7fd07bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:56.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.009+0000 7fd07bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:56.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.009+0000 7fd07bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:56.011 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.009+0000 7fd07bfff640 10 monclient(hunting): my global_id is 4341 2024-03-25T18:10:56.011 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.009+0000 7fd07bfff640 1 -- 172.21.15.156:0/4121091622 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0840b28a0 con 0x7fd08413f710 2024-03-25T18:10:56.015 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.013+0000 7fd07bfff640 1 -- 172.21.15.156:0/4121091622 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1281243947 0 0) 0x7fd070003520 con 0x7fd0841430c0 2024-03-25T18:10:56.015 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.013+0000 7fd07bfff640 1 -- 172.21.15.156:0/4121091622 >> v1:172.21.15.156:6790/0 conn(0x7fd0840a98f0 legacy=0x7fd0840a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:56.015 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.014+0000 7fd07bfff640 1 -- 172.21.15.156:0/4121091622 >> v1:172.21.15.156:6789/0 conn(0x7fd08413f710 legacy=0x7fd0841429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:56.015 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.014+0000 7fd07bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:10:56.015 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.014+0000 7fd07bfff640 1 monclient: found mon.noname-c 2024-03-25T18:10:56.015 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.014+0000 7fd07bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:56.015 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.014+0000 7fd07bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:10:56.015 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.014+0000 7fd07bfff640 1 -- 172.21.15.156:0/4121091622 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0840b44c0 con 0x7fd0841430c0 2024-03-25T18:10:56.016 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.014+0000 7fd07bfff640 10 monclient: _finish_auth 0 2024-03-25T18:10:56.016 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.014+0000 7fd07bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:10:56.016 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.014+0000 7fd07bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:56.016 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.014+0000 7fd08ba25640 5 monclient: authenticate success, global_id 4336 2024-03-25T18:10:56.016 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.014+0000 7fd07bfff640 1 -- 172.21.15.156:0/4121091622 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fd07c001a10 con 0x7fd0841430c0 2024-03-25T18:10:56.016 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.014+0000 7fd08ba25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:56.016 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.014+0000 7fd07bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:56.016 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.014+0000 7fd07bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:56.016 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.014+0000 7fd07bfff640 10 monclient: dump: 2024-03-25T18:10:56.016 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:56.016 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:56.016 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:56.016 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:56.017 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:56.017 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:56.017 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:56.017 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:56.017 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:56.017 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:56.017 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.014+0000 7fd08ba25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:56.035 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.034+0000 7fd07bfff640 1 -- 172.21.15.156:0/4121091622 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd07c001b70 con 0x7fd0841430c0 2024-03-25T18:10:56.035 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.034+0000 7fd07bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:56.035 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.034+0000 7fd07bfff640 1 -- 172.21.15.156:0/4121091622 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fd07c001f80 con 0x7fd0841430c0 2024-03-25T18:10:56.035 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.034+0000 7fd07bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:56.035 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.034+0000 7fd07bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:56.035 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.034+0000 7fd07bfff640 10 monclient: dump: 2024-03-25T18:10:56.035 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:56.035 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:56.035 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:56.036 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:56.036 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:56.036 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:56.036 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:56.036 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:56.036 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:56.036 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:56.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.034+0000 7fd08ba25640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:56.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.034+0000 7fd08ba25640 4 set_mon_vals no callback set 2024-03-25T18:10:56.039 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.035+0000 7fd08ba25640 10 monclient: shutdown 2024-03-25T18:10:56.039 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.035+0000 7fd08ba25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:56.039 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.035+0000 7fd08ba25640 1 -- 172.21.15.156:0/4121091622 >> v1:172.21.15.194:6789/0 conn(0x7fd0841430c0 legacy=0x7fd08414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:56.039 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.035+0000 7fd08ba25640 1 -- 172.21.15.156:0/4121091622 shutdown_connections 2024-03-25T18:10:56.039 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.035+0000 7fd08ba25640 1 -- 172.21.15.156:0/4121091622 >> 172.21.15.156:0/4121091622 conn(0x7fd0840b7ea0 msgr2=0x7fd0840b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:56.039 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.035+0000 7fd08ba25640 1 -- 172.21.15.156:0/4121091622 shutdown_connections 2024-03-25T18:10:56.039 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.035+0000 7fd08ba25640 1 -- 172.21.15.156:0/4121091622 wait complete. 2024-03-25T18:10:56.039 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.036+0000 7fd08ba25640 10 monclient: build_initial_monmap 2024-03-25T18:10:56.040 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.036+0000 7fd08ba25640 10 monclient: monmap: 2024-03-25T18:10:56.040 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:10:56.040 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:56.040 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:10:56.040 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:10:56.040 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:56.040 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:10:56.040 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:10:56.040 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:10:56.040 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:10:56.040 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:56.040 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.036+0000 7fd08ba25640 1 librados: starting msgr at 2024-03-25T18:10:56.040 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.036+0000 7fd08ba25640 1 librados: starting objecter 2024-03-25T18:10:56.040 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.036+0000 7fd08ba25640 1 Processor -- start 2024-03-25T18:10:56.041 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.036+0000 7fd08ba25640 1 -- start start 2024-03-25T18:10:56.041 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.036+0000 7fd08ba25640 1 librados: setting wanted keys 2024-03-25T18:10:56.041 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.036+0000 7fd08ba25640 1 librados: calling monclient init 2024-03-25T18:10:56.041 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.036+0000 7fd08ba25640 10 monclient: init 2024-03-25T18:10:56.041 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.037+0000 7fd08ba25640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:56.041 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.037+0000 7fd08ba25640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:10:56.041 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.037+0000 7fd08ba25640 10 monclient(hunting): picked mon.noname-c con 0x7fd0840a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:10:56.041 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.037+0000 7fd08ba25640 10 monclient(hunting): picked mon.noname-b con 0x7fd08413f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:10:56.041 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.037+0000 7fd08ba25640 10 monclient(hunting): picked mon.noname-a con 0x7fd0841430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:10:56.041 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.037+0000 7fd08ba25640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd084135230 con 0x7fd0841430c0 2024-03-25T18:10:56.041 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.037+0000 7fd08ba25640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd084135da0 con 0x7fd08413f710 2024-03-25T18:10:56.041 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.037+0000 7fd08ba25640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd084135f30 con 0x7fd0840a98f0 2024-03-25T18:10:56.041 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.037+0000 7fd08ba25640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:56.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.037+0000 7fd08ba25640 10 monclient(hunting): authenticate will time out at 7137.348484s 2024-03-25T18:10:56.051 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.050+0000 7fd089fc2640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fd0841430c0 0x7fd0841db560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:54680/0 (socket says 172.21.15.156:54680) 2024-03-25T18:10:56.051 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.051+0000 7fd089fc2640 1 -- 172.21.15.156:0/3751389639 learned_addr learned my addr 172.21.15.156:0/3751389639 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:10:56.054 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.053+0000 7fd079ffb640 10 client.?.objecter ms_handle_connect 0x7fd0841430c0 2024-03-25T18:10:56.054 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.053+0000 7fd079ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:56.056 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.056+0000 7fd079ffb640 10 client.?.objecter ms_handle_connect 0x7fd0840a98f0 2024-03-25T18:10:56.056 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.056+0000 7fd079ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:56.057 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.056+0000 7fd079ffb640 10 client.?.objecter ms_handle_connect 0x7fd08413f710 2024-03-25T18:10:56.057 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.056+0000 7fd079ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:56.073 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.072+0000 7fd079ffb640 1 -- 172.21.15.156:0/3751389639 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4109514219 0 0) 0x7fd084135f30 con 0x7fd0840a98f0 2024-03-25T18:10:56.073 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.072+0000 7fd079ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:56.073 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.072+0000 7fd079ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:56.073 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.072+0000 7fd079ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:56.073 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.072+0000 7fd079ffb640 10 monclient(hunting): my global_id is 4349 2024-03-25T18:10:56.073 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.072+0000 7fd079ffb640 1 -- 172.21.15.156:0/3751389639 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd068004eb0 con 0x7fd0840a98f0 2024-03-25T18:10:56.075 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.074+0000 7fd079ffb640 1 -- 172.21.15.156:0/3751389639 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2601264572 0 0) 0x7fd084135230 con 0x7fd0841430c0 2024-03-25T18:10:56.075 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.074+0000 7fd079ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:56.075 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.074+0000 7fd079ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:56.075 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.074+0000 7fd079ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:56.075 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.074+0000 7fd079ffb640 10 monclient(hunting): my global_id is 4344 2024-03-25T18:10:56.075 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.074+0000 7fd079ffb640 1 -- 172.21.15.156:0/3751389639 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd084135f30 con 0x7fd0841430c0 2024-03-25T18:10:56.075 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.074+0000 7fd079ffb640 1 -- 172.21.15.156:0/3751389639 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031251217 0 0) 0x7fd084135da0 con 0x7fd08413f710 2024-03-25T18:10:56.075 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.074+0000 7fd079ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:56.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.074+0000 7fd079ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:56.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.074+0000 7fd079ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:56.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.074+0000 7fd079ffb640 10 monclient(hunting): my global_id is 4339 2024-03-25T18:10:56.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.074+0000 7fd079ffb640 1 -- 172.21.15.156:0/3751389639 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd084135230 con 0x7fd08413f710 2024-03-25T18:10:56.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.086+0000 7fd079ffb640 1 -- 172.21.15.156:0/3751389639 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3877256242 0 0) 0x7fd084135230 con 0x7fd08413f710 2024-03-25T18:10:56.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.086+0000 7fd079ffb640 1 -- 172.21.15.156:0/3751389639 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd084135da0 con 0x7fd08413f710 2024-03-25T18:10:56.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.086+0000 7fd079ffb640 1 -- 172.21.15.156:0/3751389639 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fd074001930 con 0x7fd08413f710 2024-03-25T18:10:56.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.086+0000 7fd079ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:56.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.086+0000 7fd079ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:56.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.086+0000 7fd079ffb640 10 monclient(hunting): dump: 2024-03-25T18:10:56.088 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:56.088 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:56.089 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:56.089 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:56.089 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:56.089 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:56.089 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:56.089 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:56.089 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:56.089 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:56.089 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.086+0000 7fd079ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:56.090 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.088+0000 7fd079ffb640 1 -- 172.21.15.156:0/3751389639 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1633428490 0 0) 0x7fd068004eb0 con 0x7fd0840a98f0 2024-03-25T18:10:56.090 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.088+0000 7fd079ffb640 1 -- 172.21.15.156:0/3751389639 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd084135230 con 0x7fd0840a98f0 2024-03-25T18:10:56.090 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.088+0000 7fd079ffb640 1 -- 172.21.15.156:0/3751389639 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fd0800018c0 con 0x7fd0840a98f0 2024-03-25T18:10:56.090 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.088+0000 7fd079ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:56.090 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.088+0000 7fd079ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:56.090 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.088+0000 7fd079ffb640 10 monclient(hunting): dump: 2024-03-25T18:10:56.090 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:56.090 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:56.090 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:56.090 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:56.090 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:56.091 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:56.091 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:56.091 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:56.091 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:56.091 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:56.096 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.095+0000 7fd079ffb640 1 -- 172.21.15.156:0/3751389639 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3961503625 0 0) 0x7fd084135da0 con 0x7fd08413f710 2024-03-25T18:10:56.096 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.095+0000 7fd079ffb640 1 -- 172.21.15.156:0/3751389639 >> v1:172.21.15.156:6790/0 conn(0x7fd0840a98f0 legacy=0x7fd0841d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:56.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.095+0000 7fd079ffb640 1 -- 172.21.15.156:0/3751389639 >> v1:172.21.15.156:6789/0 conn(0x7fd0841430c0 legacy=0x7fd0841db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:56.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.095+0000 7fd079ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:10:56.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.095+0000 7fd079ffb640 1 monclient: found mon.b 2024-03-25T18:10:56.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.095+0000 7fd079ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:56.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.095+0000 7fd079ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:10:56.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.095+0000 7fd079ffb640 1 -- 172.21.15.156:0/3751389639 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0841360c0 con 0x7fd08413f710 2024-03-25T18:10:56.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.095+0000 7fd079ffb640 10 monclient: _finish_auth 0 2024-03-25T18:10:56.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.095+0000 7fd079ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:10:56.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.095+0000 7fd079ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:56.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.095+0000 7fd08ba25640 5 monclient: authenticate success, global_id 4339 2024-03-25T18:10:56.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.095+0000 7fd08ba25640 10 monclient: _renew_subs 2024-03-25T18:10:56.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.095+0000 7fd08ba25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:10:56.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.095+0000 7fd08ba25640 1 -- 172.21.15.156:0/3751389639 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0841d5000 con 0x7fd08413f710 2024-03-25T18:10:56.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.095+0000 7fd08ba25640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:56.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.095+0000 7fd08ba25640 10 monclient: _renew_subs 2024-03-25T18:10:56.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.095+0000 7fd08ba25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:10:56.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.095+0000 7fd08ba25640 1 -- 172.21.15.156:0/3751389639 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0841d41e0 con 0x7fd08413f710 2024-03-25T18:10:56.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.095+0000 7fd08ba25640 1 librados: init done 2024-03-25T18:10:56.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.097+0000 7fd08ba25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:56.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.097+0000 7fd08ba25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:56.099 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.097+0000 7fd08ba25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:10:56.099 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.097+0000 7fd08ba25640 1 -- 172.21.15.156:0/3751389639 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd084135090 con 0x7fd08413f710 2024-03-25T18:10:56.119 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.118+0000 7fd079ffb640 1 -- 172.21.15.156:0/3751389639 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd074003610 con 0x7fd08413f710 2024-03-25T18:10:56.119 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.118+0000 7fd079ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:56.119 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.118+0000 7fd079ffb640 1 -- 172.21.15.156:0/3751389639 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fd074003780 con 0x7fd08413f710 2024-03-25T18:10:56.119 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.118+0000 7fd079ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:56.119 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.118+0000 7fd079ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:56.119 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.118+0000 7fd079ffb640 10 monclient: dump: 2024-03-25T18:10:56.119 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:10:56.119 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:10:56.120 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:56.120 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:10:56.120 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:56.120 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:10:56.120 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:10:56.120 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:10:56.120 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:10:56.120 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:10:56.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.118+0000 7fd07b7fe640 4 set_mon_vals no callback set 2024-03-25T18:10:56.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.119+0000 7fd079ffb640 1 -- 172.21.15.156:0/3751389639 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7fd074019750 con 0x7fd08413f710 2024-03-25T18:10:56.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.119+0000 7fd0897c1640 10 monclient: get_auth_request con 0x7fd06806df70 auth_method 0 2024-03-25T18:10:56.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.120+0000 7fd079ffb640 10 client.4339.objecter ms_handle_connect 0x7fd06806df70 2024-03-25T18:10:56.123 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.122+0000 7fd079ffb640 1 -- 172.21.15.156:0/3751389639 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7fd074053380 con 0x7fd08413f710 2024-03-25T18:10:56.123 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.122+0000 7fd079ffb640 10 client.4339.objecter ms_dispatch 0x7fd084000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:56.123 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.122+0000 7fd079ffb640 3 client.4339.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:56.123 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.122+0000 7fd079ffb640 3 client.4339.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:56.123 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.122+0000 7fd079ffb640 20 client.4339.objecter dump_active .. 0 homeless 2024-03-25T18:10:56.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.380+0000 7fd079ffb640 1 -- 172.21.15.156:0/3751389639 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd07404ee90 con 0x7fd08413f710 2024-03-25T18:10:56.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.380+0000 7fd079ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:56.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.380+0000 7fd079ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:56.614 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.613+0000 7fd08ba25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:56.614 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.613+0000 7fd08ba25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:56.614 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.613+0000 7fd08ba25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:10:56.614 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.613+0000 7fd08ba25640 1 -- 172.21.15.156:0/3751389639 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0841353c0 con 0x7fd08413f710 2024-03-25T18:10:56.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.637+0000 7fd079ffb640 1 -- 172.21.15.156:0/3751389639 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7fd074050dc0 con 0x7fd08413f710 2024-03-25T18:10:56.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.637+0000 7fd079ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:56.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.637+0000 7fd079ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:56.638 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:10:56.638 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:56.641 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.640+0000 7fd08ba25640 10 librados: watch_flush enter 2024-03-25T18:10:56.641 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.640+0000 7fd08ba25640 10 librados: watch_flush exit 2024-03-25T18:10:56.641 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.640+0000 7fd08ba25640 20 client.4339.objecter shutdown clearing up homeless session... 2024-03-25T18:10:56.642 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.640+0000 7fd08ba25640 10 client.4339.objecter successfully canceled tick 2024-03-25T18:10:56.642 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.640+0000 7fd08ba25640 1 -- 172.21.15.156:0/3751389639 >> v1:172.21.15.194:6800/2946427713 conn(0x7fd06806df70 legacy=0x7fd068070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:56.642 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.640+0000 7fd08ba25640 10 monclient: shutdown 2024-03-25T18:10:56.642 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.640+0000 7fd08ba25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:56.642 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.640+0000 7fd08ba25640 1 -- 172.21.15.156:0/3751389639 >> v1:172.21.15.194:6789/0 conn(0x7fd08413f710 legacy=0x7fd0841dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:56.642 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.640+0000 7fd08ba25640 1 -- 172.21.15.156:0/3751389639 shutdown_connections 2024-03-25T18:10:56.642 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.640+0000 7fd08ba25640 1 -- 172.21.15.156:0/3751389639 >> 172.21.15.156:0/3751389639 conn(0x7fd0840b7ea0 msgr2=0x7fd08414ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:56.642 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.640+0000 7fd08ba25640 1 -- 172.21.15.156:0/3751389639 shutdown_connections 2024-03-25T18:10:56.642 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.640+0000 7fd08ba25640 1 -- 172.21.15.156:0/3751389639 wait complete. 2024-03-25T18:10:56.642 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:10:56.640+0000 7fd08ba25640 1 librados: shutdown 2024-03-25T18:10:56.653 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:03.460 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:03.577 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.576+0000 7f070046e640 10 monclient: get_monmap_and_config 2024-03-25T18:11:03.577 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.576+0000 7f070046e640 10 monclient: build_initial_monmap 2024-03-25T18:11:03.577 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.576+0000 7f070046e640 10 monclient: monmap: 2024-03-25T18:11:03.577 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:11:03.577 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:03.577 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:11:03.577552+0000 2024-03-25T18:11:03.577 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:11:03.577552+0000 2024-03-25T18:11:03.577 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:03.578 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:03.578 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:11:03.578 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:11:03.578 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:11:03.578 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:03.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.577+0000 7f070046e640 1 Processor -- start 2024-03-25T18:11:03.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.577+0000 7f070046e640 1 -- start start 2024-03-25T18:11:03.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.577+0000 7f070046e640 10 monclient: init 2024-03-25T18:11:03.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.578+0000 7f070046e640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:03.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.578+0000 7f070046e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:11:03.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.578+0000 7f070046e640 10 monclient(hunting): picked mon.noname-c con 0x7f06f80b4d60 addr v1:172.21.15.194:6789/0 2024-03-25T18:11:03.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.578+0000 7f070046e640 10 monclient(hunting): picked mon.noname-b con 0x7f06f80b1730 addr v1:172.21.15.156:6790/0 2024-03-25T18:11:03.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.578+0000 7f070046e640 10 monclient(hunting): picked mon.noname-a con 0x7f06f814be90 addr v1:172.21.15.156:6789/0 2024-03-25T18:11:03.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.578+0000 7f070046e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f80af940 con 0x7f06f814be90 2024-03-25T18:11:03.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.578+0000 7f070046e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f80b0a70 con 0x7f06f80b4d60 2024-03-25T18:11:03.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.578+0000 7f070046e640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f80b0d50 con 0x7f06f80b1730 2024-03-25T18:11:03.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.578+0000 7f070046e640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:03.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.578+0000 7f070046e640 10 monclient(hunting): authenticate will time out at 7144.889187s 2024-03-25T18:11:03.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.578+0000 7f06fda09640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f06f80b1730 0x7f06f80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:60960/0 (socket says 172.21.15.156:60960) 2024-03-25T18:11:03.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.578+0000 7f06fda09640 1 -- 172.21.15.156:0/2713614286 learned_addr learned my addr 172.21.15.156:0/2713614286 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:11:03.606 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:03:09.596 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:03.613 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:03:09.604 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:03.642 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:03:09.633 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:03.648 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.648+0000 7f06fd208640 1 -- 172.21.15.156:0/2713614286 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042464637 0 0) 0x7f06f80af940 con 0x7f06f814be90 2024-03-25T18:11:03.648 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.648+0000 7f06fd208640 10 monclient(hunting): _negotiate 2024-03-25T18:11:03.649 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.648+0000 7f06fd208640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:03.649 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.648+0000 7f06fd208640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:03.649 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.648+0000 7f06fd208640 10 monclient(hunting): my global_id is 4347 2024-03-25T18:11:03.649 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.648+0000 7f06fd208640 1 -- 172.21.15.156:0/2713614286 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06e4003520 con 0x7f06f814be90 2024-03-25T18:11:03.669 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:03:09.636 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:03.680 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.679+0000 7f06fd208640 1 -- 172.21.15.156:0/2713614286 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1176658326 0 0) 0x7f06f80b0a70 con 0x7f06f80b4d60 2024-03-25T18:11:03.680 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.679+0000 7f06fd208640 10 monclient(hunting): _negotiate 2024-03-25T18:11:03.680 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.679+0000 7f06fd208640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:03.680 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.679+0000 7f06fd208640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:03.680 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.679+0000 7f06fd208640 10 monclient(hunting): my global_id is 4342 2024-03-25T18:11:03.680 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.679+0000 7f06fd208640 1 -- 172.21.15.156:0/2713614286 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06f80af940 con 0x7f06f80b4d60 2024-03-25T18:11:03.681 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:03:09.647 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:03.686 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:03:09.676 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:03.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.685+0000 7f06fd208640 1 -- 172.21.15.156:0/2713614286 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971508603 0 0) 0x7f06f80b0d50 con 0x7f06f80b1730 2024-03-25T18:11:03.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.685+0000 7f06fd208640 10 monclient(hunting): _negotiate 2024-03-25T18:11:03.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.685+0000 7f06fd208640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:03.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.685+0000 7f06fd208640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:03.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.685+0000 7f06fd208640 10 monclient(hunting): my global_id is 4352 2024-03-25T18:11:03.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.685+0000 7f06fd208640 1 -- 172.21.15.156:0/2713614286 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06f80b0a70 con 0x7f06f80b1730 2024-03-25T18:11:03.689 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:03:09.680 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:03.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.709+0000 7f06fd208640 1 -- 172.21.15.156:0/2713614286 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 173023009 0 0) 0x7f06e4003520 con 0x7f06f814be90 2024-03-25T18:11:03.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.709+0000 7f06fd208640 1 -- 172.21.15.156:0/2713614286 >> v1:172.21.15.156:6790/0 conn(0x7f06f80b1730 legacy=0x7f06f80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:03.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.709+0000 7f06fd208640 1 -- 172.21.15.156:0/2713614286 >> v1:172.21.15.194:6789/0 conn(0x7f06f80b4d60 legacy=0x7f06f813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:03.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.709+0000 7f06fd208640 10 monclient: _finish_hunting 0 2024-03-25T18:11:03.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.709+0000 7f06fd208640 1 monclient: found mon.noname-a 2024-03-25T18:11:03.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.709+0000 7f06fd208640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:03.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.709+0000 7f06fd208640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:11:03.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.709+0000 7f06fd208640 1 -- 172.21.15.156:0/2713614286 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f80b1030 con 0x7f06f814be90 2024-03-25T18:11:03.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.709+0000 7f06fd208640 10 monclient: _finish_auth 0 2024-03-25T18:11:03.711 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.709+0000 7f06fd208640 10 monclient: _check_auth_tickets 2024-03-25T18:11:03.711 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.709+0000 7f06fd208640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:03.711 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.709+0000 7f06fd208640 1 -- 172.21.15.156:0/2713614286 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f06f0001a10 con 0x7f06f814be90 2024-03-25T18:11:03.711 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.709+0000 7f06fd208640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:03.711 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.709+0000 7f06fd208640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:03.711 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.709+0000 7f06fd208640 10 monclient: dump: 2024-03-25T18:11:03.711 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:03.711 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:03.711 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:03.711 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:03.711 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:03.711 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:03.711 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:03.711 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:03.711 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:03.711 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:03.712 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.709+0000 7f070046e640 5 monclient: authenticate success, global_id 4347 2024-03-25T18:11:03.712 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.709+0000 7f070046e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:03.718 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:03:09.685 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:03.725 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.724+0000 7f06fd208640 1 -- 172.21.15.156:0/2713614286 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06f0001b70 con 0x7f06f814be90 2024-03-25T18:11:03.725 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.724+0000 7f06fd208640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:03.725 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.724+0000 7f06fd208640 1 -- 172.21.15.156:0/2713614286 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f06f0001f80 con 0x7f06f814be90 2024-03-25T18:11:03.725 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.724+0000 7f06fd208640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:03.725 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.724+0000 7f06fd208640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:03.725 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.724+0000 7f06fd208640 10 monclient: dump: 2024-03-25T18:11:03.725 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:03.725 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:03.726 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:03.726 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:03.726 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:03.726 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:03.726 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:03.726 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:03.726 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:03.726 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:03.726 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.724+0000 7f070046e640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:03.726 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.724+0000 7f070046e640 4 set_mon_vals no callback set 2024-03-25T18:11:03.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.725+0000 7f070046e640 10 monclient: shutdown 2024-03-25T18:11:03.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.725+0000 7f070046e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:03.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.725+0000 7f070046e640 1 -- 172.21.15.156:0/2713614286 >> v1:172.21.15.156:6789/0 conn(0x7f06f814be90 legacy=0x7f06f814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:03.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.725+0000 7f070046e640 1 -- 172.21.15.156:0/2713614286 shutdown_connections 2024-03-25T18:11:03.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.725+0000 7f070046e640 1 -- 172.21.15.156:0/2713614286 >> 172.21.15.156:0/2713614286 conn(0x7f06f80617a0 msgr2=0x7f06f8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:03.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.725+0000 7f070046e640 1 -- 172.21.15.156:0/2713614286 shutdown_connections 2024-03-25T18:11:03.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.725+0000 7f070046e640 1 -- 172.21.15.156:0/2713614286 wait complete. 2024-03-25T18:11:03.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.726+0000 7f070046e640 10 monclient: build_initial_monmap 2024-03-25T18:11:03.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.726+0000 7f070046e640 10 monclient: monmap: 2024-03-25T18:11:03.729 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:11:03.729 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:03.729 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:11:03.729 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:11:03.729 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:03.729 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:11:03.729 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:11:03.729 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:11:03.729 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:11:03.729 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:03.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.726+0000 7f070046e640 1 librados: starting msgr at 2024-03-25T18:11:03.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.726+0000 7f070046e640 1 librados: starting objecter 2024-03-25T18:11:03.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.726+0000 7f070046e640 1 Processor -- start 2024-03-25T18:11:03.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.726+0000 7f070046e640 1 -- start start 2024-03-25T18:11:03.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.726+0000 7f070046e640 1 librados: setting wanted keys 2024-03-25T18:11:03.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.726+0000 7f070046e640 1 librados: calling monclient init 2024-03-25T18:11:03.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.726+0000 7f070046e640 10 monclient: init 2024-03-25T18:11:03.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.727+0000 7f070046e640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:03.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.727+0000 7f070046e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:11:03.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.727+0000 7f070046e640 10 monclient(hunting): picked mon.noname-b con 0x7f06f80b1730 addr v1:172.21.15.194:6789/0 2024-03-25T18:11:03.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.727+0000 7f070046e640 10 monclient(hunting): picked mon.noname-c con 0x7f06f80b4d60 addr v1:172.21.15.156:6790/0 2024-03-25T18:11:03.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.727+0000 7f070046e640 10 monclient(hunting): picked mon.noname-a con 0x7f06f814be90 addr v1:172.21.15.156:6789/0 2024-03-25T18:11:03.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.727+0000 7f070046e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f8141950 con 0x7f06f814be90 2024-03-25T18:11:03.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.727+0000 7f070046e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f81d7940 con 0x7f06f80b1730 2024-03-25T18:11:03.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.727+0000 7f070046e640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f81d8570 con 0x7f06f80b4d60 2024-03-25T18:11:03.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.727+0000 7f070046e640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:03.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.727+0000 7f070046e640 10 monclient(hunting): authenticate will time out at 7145.037994s 2024-03-25T18:11:03.735 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.735+0000 7f06fda09640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f06f80b4d60 0x7f06f8142860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:60996/0 (socket says 172.21.15.156:60996) 2024-03-25T18:11:03.735 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.735+0000 7f06fda09640 1 -- 172.21.15.156:0/2053239956 learned_addr learned my addr 172.21.15.156:0/2053239956 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:11:03.736 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:03:09.726 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:03.736 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.735+0000 7f06eeffd640 10 client.?.objecter ms_handle_connect 0x7f06f80b4d60 2024-03-25T18:11:03.736 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.735+0000 7f06eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:03.742 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.742+0000 7f06eeffd640 1 -- 172.21.15.156:0/2053239956 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3093452809 0 0) 0x7f06f81d8570 con 0x7f06f80b4d60 2024-03-25T18:11:03.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.742+0000 7f06eeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:03.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.742+0000 7f06eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:03.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.742+0000 7f06eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:03.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.742+0000 7f06eeffd640 10 monclient(hunting): my global_id is 4355 2024-03-25T18:11:03.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.742+0000 7f06eeffd640 1 -- 172.21.15.156:0/2053239956 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06dc004eb0 con 0x7f06f80b4d60 2024-03-25T18:11:03.752 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.751+0000 7f06eeffd640 1 -- 172.21.15.156:0/2053239956 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2654882800 0 0) 0x7f06dc004eb0 con 0x7f06f80b4d60 2024-03-25T18:11:03.752 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.751+0000 7f06eeffd640 1 -- 172.21.15.156:0/2053239956 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06f81d8570 con 0x7f06f80b4d60 2024-03-25T18:11:03.752 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.751+0000 7f06eeffd640 1 -- 172.21.15.156:0/2053239956 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f06e8001910 con 0x7f06f80b4d60 2024-03-25T18:11:03.752 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.751+0000 7f06eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:03.752 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.751+0000 7f06eeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:03.752 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.751+0000 7f06eeffd640 10 monclient(hunting): dump: 2024-03-25T18:11:03.752 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:03.752 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:03.752 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:03.752 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:03.752 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:03.752 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:03.752 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:03.752 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:03.753 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:03.753 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:03.753 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.751+0000 7f06eeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:03.757 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.756+0000 7f06eeffd640 10 client.?.objecter ms_handle_connect 0x7f06f814be90 2024-03-25T18:11:03.757 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.756+0000 7f06eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:03.760 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.760+0000 7f06eeffd640 10 client.?.objecter ms_handle_connect 0x7f06f80b1730 2024-03-25T18:11:03.761 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.760+0000 7f06eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:03.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.782+0000 7f06eeffd640 1 -- 172.21.15.156:0/2053239956 <== mon.2 v1:172.21.15.156:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2185992919 0 0) 0x7f06f81d8570 con 0x7f06f80b4d60 2024-03-25T18:11:03.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.782+0000 7f06eeffd640 1 -- 172.21.15.156:0/2053239956 >> v1:172.21.15.194:6789/0 conn(0x7f06f80b1730 legacy=0x7f06f81d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:03.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.782+0000 7f06eeffd640 1 -- 172.21.15.156:0/2053239956 >> v1:172.21.15.156:6789/0 conn(0x7f06f814be90 legacy=0x7f06f81e2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:03.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.782+0000 7f06eeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:11:03.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.782+0000 7f06eeffd640 1 monclient: found mon.c 2024-03-25T18:11:03.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.782+0000 7f06eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:03.784 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.782+0000 7f06eeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:11:03.784 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.782+0000 7f06eeffd640 1 -- 172.21.15.156:0/2053239956 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f8138840 con 0x7f06f80b4d60 2024-03-25T18:11:03.784 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.782+0000 7f06eeffd640 10 monclient: _finish_auth 0 2024-03-25T18:11:03.784 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.782+0000 7f06eeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:11:03.784 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.782+0000 7f06eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:03.784 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.782+0000 7f070046e640 5 monclient: authenticate success, global_id 4355 2024-03-25T18:11:03.784 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.782+0000 7f070046e640 10 monclient: _renew_subs 2024-03-25T18:11:03.784 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.782+0000 7f070046e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:11:03.784 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.782+0000 7f070046e640 1 -- 172.21.15.156:0/2053239956 --> v1:172.21.15.156:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f8142f70 con 0x7f06f80b4d60 2024-03-25T18:11:03.784 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.782+0000 7f070046e640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:03.784 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.782+0000 7f070046e640 10 monclient: _renew_subs 2024-03-25T18:11:03.784 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.782+0000 7f070046e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:11:03.784 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.782+0000 7f070046e640 1 -- 172.21.15.156:0/2053239956 --> v1:172.21.15.156:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f81432c0 con 0x7f06f80b4d60 2024-03-25T18:11:03.784 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.782+0000 7f070046e640 1 librados: init done 2024-03-25T18:11:03.785 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.783+0000 7f070046e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:03.785 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.783+0000 7f070046e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:03.785 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.783+0000 7f070046e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:11:03.785 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:03.783+0000 7f070046e640 1 -- 172.21.15.156:0/2053239956 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f81d7ad0 con 0x7f06f80b4d60 2024-03-25T18:11:04.108 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.107+0000 7f06eeffd640 1 -- 172.21.15.156:0/2053239956 <== mon.2 v1:172.21.15.156:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06e8001c50 con 0x7f06f80b4d60 2024-03-25T18:11:04.108 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.107+0000 7f06eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:04.108 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.107+0000 7f06eeffd640 1 -- 172.21.15.156:0/2053239956 <== mon.2 v1:172.21.15.156:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f06e8004c30 con 0x7f06f80b4d60 2024-03-25T18:11:04.108 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.107+0000 7f06eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:04.109 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.107+0000 7f06eeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:04.109 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.107+0000 7f06eeffd640 10 monclient: dump: 2024-03-25T18:11:04.109 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:04.109 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:04.109 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:04.109 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:04.109 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:04.109 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:04.109 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:04.109 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:04.109 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:04.109 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:04.109 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.107+0000 7f06fca07640 4 set_mon_vals no callback set 2024-03-25T18:11:04.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.108+0000 7f06eeffd640 1 -- 172.21.15.156:0/2053239956 <== mon.2 v1:172.21.15.156:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f06e8004e50 con 0x7f06f80b4d60 2024-03-25T18:11:04.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.109+0000 7f06fe20a640 10 monclient: get_auth_request con 0x7f06dc06dc30 auth_method 0 2024-03-25T18:11:04.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.109+0000 7f06eeffd640 10 client.4355.objecter ms_handle_connect 0x7f06dc06dc30 2024-03-25T18:11:04.116 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.115+0000 7f06eeffd640 1 -- 172.21.15.156:0/2053239956 <== mon.2 v1:172.21.15.156:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f06e8054950 con 0x7f06f80b4d60 2024-03-25T18:11:04.117 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.115+0000 7f06eeffd640 10 client.4355.objecter ms_dispatch 0x7f06f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:04.117 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.115+0000 7f06eeffd640 3 client.4355.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:04.117 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.115+0000 7f06eeffd640 3 client.4355.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:04.117 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.116+0000 7f06eeffd640 20 client.4355.objecter dump_active .. 0 homeless 2024-03-25T18:11:04.117 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.116+0000 7f06eeffd640 1 -- 172.21.15.156:0/2053239956 <== mon.2 v1:172.21.15.156:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f06e8050460 con 0x7f06f80b4d60 2024-03-25T18:11:04.117 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.116+0000 7f06eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:04.117 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.116+0000 7f06eeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:04.361 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.360+0000 7f070046e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:04.369 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.360+0000 7f070046e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:04.369 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.360+0000 7f070046e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:11:04.369 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.360+0000 7f070046e640 1 -- 172.21.15.156:0/2053239956 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06f81389d0 con 0x7f06f80b4d60 2024-03-25T18:11:04.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.386+0000 7f06eeffd640 1 -- 172.21.15.156:0/2053239956 <== mon.2 v1:172.21.15.156:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f06e8052390 con 0x7f06f80b4d60 2024-03-25T18:11:04.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.386+0000 7f06eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:04.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.386+0000 7f06eeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:04.387 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:11:04.387 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:04.390 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.388+0000 7f070046e640 10 librados: watch_flush enter 2024-03-25T18:11:04.390 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.388+0000 7f070046e640 10 librados: watch_flush exit 2024-03-25T18:11:04.390 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.388+0000 7f070046e640 20 client.4355.objecter shutdown clearing up homeless session... 2024-03-25T18:11:04.390 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.388+0000 7f070046e640 10 client.4355.objecter successfully canceled tick 2024-03-25T18:11:04.390 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.388+0000 7f070046e640 1 -- 172.21.15.156:0/2053239956 >> v1:172.21.15.194:6800/2946427713 conn(0x7f06dc06dc30 legacy=0x7f06dc0700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:04.390 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.388+0000 7f070046e640 10 monclient: shutdown 2024-03-25T18:11:04.390 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.388+0000 7f070046e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:04.390 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.388+0000 7f070046e640 1 -- 172.21.15.156:0/2053239956 >> v1:172.21.15.156:6790/0 conn(0x7f06f80b4d60 legacy=0x7f06f8142860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:04.390 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.388+0000 7f070046e640 1 -- 172.21.15.156:0/2053239956 shutdown_connections 2024-03-25T18:11:04.390 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.388+0000 7f070046e640 1 -- 172.21.15.156:0/2053239956 >> 172.21.15.156:0/2053239956 conn(0x7f06f80617a0 msgr2=0x7f06f80b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:04.390 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.388+0000 7f070046e640 1 -- 172.21.15.156:0/2053239956 shutdown_connections 2024-03-25T18:11:04.390 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.388+0000 7f070046e640 1 -- 172.21.15.156:0/2053239956 wait complete. 2024-03-25T18:11:04.390 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:04.388+0000 7f070046e640 1 librados: shutdown 2024-03-25T18:11:04.402 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:11.210 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:11.324 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.323+0000 7f34569c9640 10 monclient: get_monmap_and_config 2024-03-25T18:11:11.324 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.323+0000 7f34569c9640 10 monclient: build_initial_monmap 2024-03-25T18:11:11.324 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.323+0000 7f34569c9640 10 monclient: monmap: 2024-03-25T18:11:11.324 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:11:11.324 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:11.324 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:11:11.324175+0000 2024-03-25T18:11:11.324 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:11:11.324175+0000 2024-03-25T18:11:11.324 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:11.324 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:11.324 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:11:11.324 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:11:11.325 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:11:11.325 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:11.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.323+0000 7f34569c9640 1 Processor -- start 2024-03-25T18:11:11.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.324+0000 7f34569c9640 1 -- start start 2024-03-25T18:11:11.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.324+0000 7f34569c9640 10 monclient: init 2024-03-25T18:11:11.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.325+0000 7f34569c9640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:11.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.325+0000 7f34569c9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:11:11.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.325+0000 7f34569c9640 10 monclient(hunting): picked mon.noname-a con 0x7f34500a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:11:11.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.325+0000 7f34569c9640 10 monclient(hunting): picked mon.noname-b con 0x7f345013f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:11:11.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.325+0000 7f34569c9640 10 monclient(hunting): picked mon.noname-c con 0x7f34501430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:11:11.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.325+0000 7f34569c9640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3450142520 con 0x7f34500a98f0 2024-03-25T18:11:11.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.325+0000 7f34569c9640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34500b28a0 con 0x7f34501430c0 2024-03-25T18:11:11.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.325+0000 7f34569c9640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34500b2a30 con 0x7f345013f710 2024-03-25T18:11:11.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.325+0000 7f34569c9640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:11.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.325+0000 7f34569c9640 10 monclient(hunting): authenticate will time out at 7152.635920s 2024-03-25T18:11:11.338 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.338+0000 7f3454f66640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f34501430c0 0x7f345014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:35626/0 (socket says 172.21.15.156:35626) 2024-03-25T18:11:11.338 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.338+0000 7f3454f66640 1 -- 172.21.15.156:0/18563715 learned_addr learned my addr 172.21.15.156:0/18563715 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:11:11.349 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.348+0000 7f344effd640 1 -- 172.21.15.156:0/18563715 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1338258691 0 0) 0x7f34500b2a30 con 0x7f345013f710 2024-03-25T18:11:11.349 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.348+0000 7f344effd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:11.349 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.348+0000 7f344effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:11.349 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.348+0000 7f344effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:11.349 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.348+0000 7f344effd640 10 monclient(hunting): my global_id is 4358 2024-03-25T18:11:11.349 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.348+0000 7f344effd640 1 -- 172.21.15.156:0/18563715 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3438003520 con 0x7f345013f710 2024-03-25T18:11:11.353 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.353+0000 7f344effd640 1 -- 172.21.15.156:0/18563715 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4058172136 0 0) 0x7f34500b28a0 con 0x7f34501430c0 2024-03-25T18:11:11.353 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.353+0000 7f344effd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:11.353 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.353+0000 7f344effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:11.354 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.353+0000 7f344effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:11.354 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.353+0000 7f344effd640 10 monclient(hunting): my global_id is 4348 2024-03-25T18:11:11.354 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.353+0000 7f344effd640 1 -- 172.21.15.156:0/18563715 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34500b2a30 con 0x7f34501430c0 2024-03-25T18:11:11.358 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.357+0000 7f344effd640 1 -- 172.21.15.156:0/18563715 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 766404278 0 0) 0x7f3438003520 con 0x7f345013f710 2024-03-25T18:11:11.359 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.357+0000 7f344effd640 1 -- 172.21.15.156:0/18563715 >> v1:172.21.15.194:6789/0 conn(0x7f34501430c0 legacy=0x7f345014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:11.359 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.357+0000 7f344effd640 1 -- 172.21.15.156:0/18563715 >> v1:172.21.15.156:6789/0 conn(0x7f34500a98f0 legacy=0x7f34500a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:11:11.359 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.357+0000 7f344effd640 10 monclient: _finish_hunting 0 2024-03-25T18:11:11.359 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.357+0000 7f344effd640 1 monclient: found mon.noname-b 2024-03-25T18:11:11.359 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.357+0000 7f344effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:11.359 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.357+0000 7f344effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.156:6790/0 2024-03-25T18:11:11.359 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.357+0000 7f344effd640 1 -- 172.21.15.156:0/18563715 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34500b44c0 con 0x7f345013f710 2024-03-25T18:11:11.359 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.357+0000 7f344effd640 10 monclient: _finish_auth 0 2024-03-25T18:11:11.359 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.357+0000 7f344effd640 10 monclient: _check_auth_tickets 2024-03-25T18:11:11.359 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.357+0000 7f344effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:11.359 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.357+0000 7f34569c9640 5 monclient: authenticate success, global_id 4358 2024-03-25T18:11:11.359 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.357+0000 7f34569c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:11.364 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.363+0000 7f344effd640 1 -- 172.21.15.156:0/18563715 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f3444001650 con 0x7f345013f710 2024-03-25T18:11:11.364 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.364+0000 7f344effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:11.364 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.364+0000 7f344effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:11.364 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.364+0000 7f344effd640 10 monclient: dump: 2024-03-25T18:11:11.365 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:11.365 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:11.365 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:11.365 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:11.365 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:11.365 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:11.365 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:11.365 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:11.365 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:11.365 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:11.365 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.364+0000 7f34569c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:11.366 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.365+0000 7f344effd640 1 -- 172.21.15.156:0/18563715 <== mon.2 v1:172.21.15.156:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34440018a0 con 0x7f345013f710 2024-03-25T18:11:11.366 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.365+0000 7f344effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:11.366 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.365+0000 7f344effd640 1 -- 172.21.15.156:0/18563715 <== mon.2 v1:172.21.15.156:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f3444001c30 con 0x7f345013f710 2024-03-25T18:11:11.367 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.365+0000 7f344effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:11.367 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.365+0000 7f344effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:11.367 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.365+0000 7f344effd640 10 monclient: dump: 2024-03-25T18:11:11.367 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:11.367 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:11.367 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:11.367 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:11.367 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:11.367 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:11.367 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:11.367 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:11.367 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:11.367 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:11.367 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.365+0000 7f34569c9640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:11.368 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.365+0000 7f34569c9640 4 set_mon_vals no callback set 2024-03-25T18:11:11.369 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.366+0000 7f34569c9640 10 monclient: shutdown 2024-03-25T18:11:11.369 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.366+0000 7f34569c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:11.369 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.366+0000 7f34569c9640 1 -- 172.21.15.156:0/18563715 >> v1:172.21.15.156:6790/0 conn(0x7f345013f710 legacy=0x7f34501429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:11.369 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.367+0000 7f34569c9640 1 -- 172.21.15.156:0/18563715 shutdown_connections 2024-03-25T18:11:11.369 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.367+0000 7f34569c9640 1 -- 172.21.15.156:0/18563715 >> 172.21.15.156:0/18563715 conn(0x7f34500b7ea0 msgr2=0x7f34500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:11.369 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.367+0000 7f34569c9640 1 -- 172.21.15.156:0/18563715 shutdown_connections 2024-03-25T18:11:11.369 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.367+0000 7f34569c9640 1 -- 172.21.15.156:0/18563715 wait complete. 2024-03-25T18:11:11.369 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.368+0000 7f34569c9640 10 monclient: build_initial_monmap 2024-03-25T18:11:11.369 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.368+0000 7f34569c9640 10 monclient: monmap: 2024-03-25T18:11:11.369 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:11:11.369 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:11.369 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:11:11.369 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:11:11.370 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:11.370 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:11:11.370 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:11:11.370 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:11:11.370 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:11:11.370 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:11.370 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.368+0000 7f34569c9640 1 librados: starting msgr at 2024-03-25T18:11:11.370 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.368+0000 7f34569c9640 1 librados: starting objecter 2024-03-25T18:11:11.370 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.368+0000 7f34569c9640 1 Processor -- start 2024-03-25T18:11:11.371 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.368+0000 7f34569c9640 1 -- start start 2024-03-25T18:11:11.371 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.368+0000 7f34569c9640 1 librados: setting wanted keys 2024-03-25T18:11:11.371 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.368+0000 7f34569c9640 1 librados: calling monclient init 2024-03-25T18:11:11.371 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.368+0000 7f34569c9640 10 monclient: init 2024-03-25T18:11:11.371 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.369+0000 7f34569c9640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:11.371 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.369+0000 7f34569c9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:11:11.371 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.369+0000 7f34569c9640 10 monclient(hunting): picked mon.noname-c con 0x7f34500a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:11:11.371 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.369+0000 7f34569c9640 10 monclient(hunting): picked mon.noname-a con 0x7f345013f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:11:11.371 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.369+0000 7f34569c9640 10 monclient(hunting): picked mon.noname-b con 0x7f34501430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:11:11.372 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.369+0000 7f34569c9640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3450139180 con 0x7f345013f710 2024-03-25T18:11:11.372 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.369+0000 7f34569c9640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34500abb30 con 0x7f34501430c0 2024-03-25T18:11:11.372 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.369+0000 7f34569c9640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34500abda0 con 0x7f34500a98f0 2024-03-25T18:11:11.372 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.369+0000 7f34569c9640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:11.372 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.369+0000 7f34569c9640 10 monclient(hunting): authenticate will time out at 7152.680335s 2024-03-25T18:11:11.379 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.379+0000 7f3454f66640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f34501430c0 0x7f34501e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:35666/0 (socket says 172.21.15.156:35666) 2024-03-25T18:11:11.379 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.379+0000 7f3454f66640 1 -- 172.21.15.156:0/2358165648 learned_addr learned my addr 172.21.15.156:0/2358165648 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:11:11.383 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.382+0000 7f344cff9640 10 client.?.objecter ms_handle_connect 0x7f34501430c0 2024-03-25T18:11:11.383 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.382+0000 7f344cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:11.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.390+0000 7f344cff9640 10 client.?.objecter ms_handle_connect 0x7f34500a98f0 2024-03-25T18:11:11.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.390+0000 7f344cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:11.392 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.392+0000 7f344cff9640 10 client.?.objecter ms_handle_connect 0x7f345013f710 2024-03-25T18:11:11.392 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.392+0000 7f344cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:11.394 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.393+0000 7f344cff9640 1 -- 172.21.15.156:0/2358165648 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033817081 0 0) 0x7f34500abb30 con 0x7f34501430c0 2024-03-25T18:11:11.394 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.393+0000 7f344cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:11:11.394 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.393+0000 7f344cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:11.394 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.393+0000 7f344cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:11.394 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.393+0000 7f344cff9640 10 monclient(hunting): my global_id is 4351 2024-03-25T18:11:11.394 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.393+0000 7f344cff9640 1 -- 172.21.15.156:0/2358165648 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f343c004eb0 con 0x7f34501430c0 2024-03-25T18:11:11.401 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.400+0000 7f344cff9640 1 -- 172.21.15.156:0/2358165648 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2046215426 0 0) 0x7f34500abda0 con 0x7f34500a98f0 2024-03-25T18:11:11.401 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.400+0000 7f344cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:11:11.401 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.400+0000 7f344cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:11.401 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.400+0000 7f344cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:11.401 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.400+0000 7f344cff9640 10 monclient(hunting): my global_id is 4361 2024-03-25T18:11:11.401 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.400+0000 7f344cff9640 1 -- 172.21.15.156:0/2358165648 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34500abb30 con 0x7f34500a98f0 2024-03-25T18:11:11.407 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.406+0000 7f344cff9640 1 -- 172.21.15.156:0/2358165648 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4051947670 0 0) 0x7f343c004eb0 con 0x7f34501430c0 2024-03-25T18:11:11.407 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.406+0000 7f344cff9640 1 -- 172.21.15.156:0/2358165648 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34500abda0 con 0x7f34501430c0 2024-03-25T18:11:11.407 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.406+0000 7f344cff9640 1 -- 172.21.15.156:0/2358165648 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f3448001c10 con 0x7f34501430c0 2024-03-25T18:11:11.407 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.406+0000 7f344cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:11.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.406+0000 7f344cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:11.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.406+0000 7f344cff9640 10 monclient(hunting): dump: 2024-03-25T18:11:11.408 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:11.408 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:11.408 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:11.408 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:11.408 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:11.408 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:11.408 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:11.408 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:11.408 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:11.408 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:11.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.406+0000 7f344cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:11.414 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.413+0000 7f344cff9640 1 -- 172.21.15.156:0/2358165648 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 891886842 0 0) 0x7f34500abda0 con 0x7f34501430c0 2024-03-25T18:11:11.414 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.413+0000 7f344cff9640 1 -- 172.21.15.156:0/2358165648 >> v1:172.21.15.156:6790/0 conn(0x7f34500a98f0 legacy=0x7f34500ac4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:11.414 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.413+0000 7f344cff9640 1 -- 172.21.15.156:0/2358165648 >> v1:172.21.15.156:6789/0 conn(0x7f345013f710 legacy=0x7f3450136580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:11.414 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.413+0000 7f344cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:11:11.414 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.413+0000 7f344cff9640 1 monclient: found mon.b 2024-03-25T18:11:11.414 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.413+0000 7f344cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:11.414 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.413+0000 7f344cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:11:11.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.413+0000 7f344cff9640 1 -- 172.21.15.156:0/2358165648 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34500ac080 con 0x7f34501430c0 2024-03-25T18:11:11.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.413+0000 7f344cff9640 10 monclient: _finish_auth 0 2024-03-25T18:11:11.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.413+0000 7f344cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:11:11.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.413+0000 7f344cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:11.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.413+0000 7f34569c9640 5 monclient: authenticate success, global_id 4351 2024-03-25T18:11:11.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.413+0000 7f34569c9640 10 monclient: _renew_subs 2024-03-25T18:11:11.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.413+0000 7f34569c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:11:11.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.413+0000 7f34569c9640 1 -- 172.21.15.156:0/2358165648 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3450137d90 con 0x7f34501430c0 2024-03-25T18:11:11.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.413+0000 7f34569c9640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:11.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.413+0000 7f34569c9640 10 monclient: _renew_subs 2024-03-25T18:11:11.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.413+0000 7f34569c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:11:11.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.413+0000 7f34569c9640 1 -- 172.21.15.156:0/2358165648 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3450138b60 con 0x7f34501430c0 2024-03-25T18:11:11.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.413+0000 7f34569c9640 1 librados: init done 2024-03-25T18:11:11.416 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.414+0000 7f34569c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:11.416 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.414+0000 7f34569c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:11.416 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.414+0000 7f34569c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:11:11.416 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.414+0000 7f34569c9640 1 -- 172.21.15.156:0/2358165648 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f345014f2d0 con 0x7f34501430c0 2024-03-25T18:11:11.693 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.692+0000 7f344cff9640 1 -- 172.21.15.156:0/2358165648 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3448004b60 con 0x7f34501430c0 2024-03-25T18:11:11.693 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.692+0000 7f344cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:11.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.692+0000 7f344cff9640 1 -- 172.21.15.156:0/2358165648 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f3448004e70 con 0x7f34501430c0 2024-03-25T18:11:11.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.692+0000 7f344cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:11.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.692+0000 7f344cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:11.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.692+0000 7f344cff9640 10 monclient: dump: 2024-03-25T18:11:11.694 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:11.694 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:11.694 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:11.694 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:11.694 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:11.694 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:11.695 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:11.695 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:11.695 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:11.695 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:11.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.692+0000 7f344e7fc640 4 set_mon_vals no callback set 2024-03-25T18:11:11.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.693+0000 7f344cff9640 1 -- 172.21.15.156:0/2358165648 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f344801ae40 con 0x7f34501430c0 2024-03-25T18:11:11.696 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.694+0000 7f344ffff640 10 monclient: get_auth_request con 0x7f343c06dd00 auth_method 0 2024-03-25T18:11:11.696 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.694+0000 7f344cff9640 10 client.4351.objecter ms_handle_connect 0x7f343c06dd00 2024-03-25T18:11:11.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.699+0000 7f344cff9640 1 -- 172.21.15.156:0/2358165648 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f3448054bc0 con 0x7f34501430c0 2024-03-25T18:11:11.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.699+0000 7f344cff9640 10 client.4351.objecter ms_dispatch 0x7f3450000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:11.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.699+0000 7f344cff9640 3 client.4351.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:11.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.699+0000 7f344cff9640 3 client.4351.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:11.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.699+0000 7f344cff9640 20 client.4351.objecter dump_active .. 0 homeless 2024-03-25T18:11:11.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.699+0000 7f344cff9640 1 -- 172.21.15.156:0/2358165648 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3448050500 con 0x7f34501430c0 2024-03-25T18:11:11.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.699+0000 7f344cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:11.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.699+0000 7f344cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:11.947 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.946+0000 7f34569c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:11.948 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.946+0000 7f34569c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:11.948 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.946+0000 7f34569c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:11:11.948 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.946+0000 7f34569c9640 1 -- 172.21.15.156:0/2358165648 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34500627b0 con 0x7f34501430c0 2024-03-25T18:11:11.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.976+0000 7f344cff9640 1 -- 172.21.15.156:0/2358165648 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f3448052600 con 0x7f34501430c0 2024-03-25T18:11:11.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.976+0000 7f344cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:11.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.976+0000 7f344cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:11.978 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:11:11.978 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:11.980 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.979+0000 7f34569c9640 10 librados: watch_flush enter 2024-03-25T18:11:11.980 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.979+0000 7f34569c9640 10 librados: watch_flush exit 2024-03-25T18:11:11.980 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.979+0000 7f34569c9640 20 client.4351.objecter shutdown clearing up homeless session... 2024-03-25T18:11:11.981 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.979+0000 7f34569c9640 10 client.4351.objecter successfully canceled tick 2024-03-25T18:11:11.981 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.979+0000 7f34569c9640 1 -- 172.21.15.156:0/2358165648 >> v1:172.21.15.194:6800/2946427713 conn(0x7f343c06dd00 legacy=0x7f343c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:11.981 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.979+0000 7f34569c9640 10 monclient: shutdown 2024-03-25T18:11:11.981 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.979+0000 7f34569c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:11.981 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.979+0000 7f34569c9640 1 -- 172.21.15.156:0/2358165648 >> v1:172.21.15.194:6789/0 conn(0x7f34501430c0 legacy=0x7f34501e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:11.981 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.979+0000 7f34569c9640 1 -- 172.21.15.156:0/2358165648 shutdown_connections 2024-03-25T18:11:11.981 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.979+0000 7f34569c9640 1 -- 172.21.15.156:0/2358165648 >> 172.21.15.156:0/2358165648 conn(0x7f34500b7ea0 msgr2=0x7f3450074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:11.981 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.979+0000 7f34569c9640 1 -- 172.21.15.156:0/2358165648 shutdown_connections 2024-03-25T18:11:11.981 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.979+0000 7f34569c9640 1 -- 172.21.15.156:0/2358165648 wait complete. 2024-03-25T18:11:11.981 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:11.979+0000 7f34569c9640 1 librados: shutdown 2024-03-25T18:11:11.992 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:18.799 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:18.915 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.915+0000 7f75a764e640 10 monclient: get_monmap_and_config 2024-03-25T18:11:18.916 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.915+0000 7f75a764e640 10 monclient: build_initial_monmap 2024-03-25T18:11:18.916 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.915+0000 7f75a764e640 10 monclient: monmap: 2024-03-25T18:11:18.916 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:11:18.916 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:18.916 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:11:18.916057+0000 2024-03-25T18:11:18.916 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:11:18.916057+0000 2024-03-25T18:11:18.916 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:18.916 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:18.916 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:11:18.916 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:11:18.916 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:11:18.917 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:18.918 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.915+0000 7f75a764e640 1 Processor -- start 2024-03-25T18:11:18.918 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.916+0000 7f75a764e640 1 -- start start 2024-03-25T18:11:18.918 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.916+0000 7f75a764e640 10 monclient: init 2024-03-25T18:11:18.918 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.916+0000 7f75a764e640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:18.918 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.916+0000 7f75a764e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:11:18.918 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.916+0000 7f75a764e640 10 monclient(hunting): picked mon.noname-a con 0x7f75a00a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:11:18.918 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.916+0000 7f75a764e640 10 monclient(hunting): picked mon.noname-b con 0x7f75a013f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:11:18.919 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.916+0000 7f75a764e640 10 monclient(hunting): picked mon.noname-c con 0x7f75a01430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:11:18.919 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.916+0000 7f75a764e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a0142520 con 0x7f75a00a98f0 2024-03-25T18:11:18.919 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.916+0000 7f75a764e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a00b28a0 con 0x7f75a01430c0 2024-03-25T18:11:18.919 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.916+0000 7f75a764e640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a00b2a30 con 0x7f75a013f710 2024-03-25T18:11:18.919 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.916+0000 7f75a764e640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:18.919 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.916+0000 7f75a764e640 10 monclient(hunting): authenticate will time out at 7160.227553s 2024-03-25T18:11:18.929 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.929+0000 7f75a5beb640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f75a01430c0 0x7f75a014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:58062/0 (socket says 172.21.15.156:58062) 2024-03-25T18:11:18.929 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.929+0000 7f75a5beb640 1 -- 172.21.15.156:0/314097771 learned_addr learned my addr 172.21.15.156:0/314097771 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:11:18.944 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.943+0000 7f7597fff640 1 -- 172.21.15.156:0/314097771 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653110337 0 0) 0x7f75a00b28a0 con 0x7f75a01430c0 2024-03-25T18:11:18.944 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.943+0000 7f7597fff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:18.944 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.943+0000 7f7597fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:18.944 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.943+0000 7f7597fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:18.944 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.943+0000 7f7597fff640 10 monclient(hunting): my global_id is 4354 2024-03-25T18:11:18.944 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.943+0000 7f7597fff640 1 -- 172.21.15.156:0/314097771 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f758c003520 con 0x7f75a01430c0 2024-03-25T18:11:18.950 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.950+0000 7f7597fff640 1 -- 172.21.15.156:0/314097771 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 431220672 0 0) 0x7f75a00b2a30 con 0x7f75a013f710 2024-03-25T18:11:18.951 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.950+0000 7f7597fff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:18.951 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.950+0000 7f7597fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:18.951 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.950+0000 7f7597fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:18.951 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.950+0000 7f7597fff640 10 monclient(hunting): my global_id is 4364 2024-03-25T18:11:18.951 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.950+0000 7f7597fff640 1 -- 172.21.15.156:0/314097771 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75a00b28a0 con 0x7f75a013f710 2024-03-25T18:11:18.953 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.952+0000 7f7597fff640 1 -- 172.21.15.156:0/314097771 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1227185036 0 0) 0x7f75a0142520 con 0x7f75a00a98f0 2024-03-25T18:11:18.953 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.952+0000 7f7597fff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:18.953 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.952+0000 7f7597fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:18.953 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.952+0000 7f7597fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:18.953 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.952+0000 7f7597fff640 10 monclient(hunting): my global_id is 4356 2024-03-25T18:11:18.953 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.952+0000 7f7597fff640 1 -- 172.21.15.156:0/314097771 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75a00b2a30 con 0x7f75a00a98f0 2024-03-25T18:11:18.960 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.959+0000 7f7597fff640 1 -- 172.21.15.156:0/314097771 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4063263431 0 0) 0x7f758c003520 con 0x7f75a01430c0 2024-03-25T18:11:18.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.959+0000 7f7597fff640 1 -- 172.21.15.156:0/314097771 >> v1:172.21.15.156:6790/0 conn(0x7f75a013f710 legacy=0x7f75a01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:18.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.959+0000 7f7597fff640 1 -- 172.21.15.156:0/314097771 >> v1:172.21.15.156:6789/0 conn(0x7f75a00a98f0 legacy=0x7f75a00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:18.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.959+0000 7f7597fff640 10 monclient: _finish_hunting 0 2024-03-25T18:11:18.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.959+0000 7f7597fff640 1 monclient: found mon.noname-c 2024-03-25T18:11:18.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.959+0000 7f7597fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:18.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.959+0000 7f7597fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:11:18.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.959+0000 7f7597fff640 1 -- 172.21.15.156:0/314097771 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a00b44c0 con 0x7f75a01430c0 2024-03-25T18:11:18.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.959+0000 7f7597fff640 10 monclient: _finish_auth 0 2024-03-25T18:11:18.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.959+0000 7f7597fff640 10 monclient: _check_auth_tickets 2024-03-25T18:11:18.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.959+0000 7f7597fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:18.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.959+0000 7f7597fff640 1 -- 172.21.15.156:0/314097771 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7598001a10 con 0x7f75a01430c0 2024-03-25T18:11:18.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.959+0000 7f75a764e640 5 monclient: authenticate success, global_id 4354 2024-03-25T18:11:18.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.959+0000 7f75a764e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:18.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.959+0000 7f7597fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:18.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.959+0000 7f7597fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:18.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.959+0000 7f7597fff640 10 monclient: dump: 2024-03-25T18:11:18.962 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:18.962 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:18.962 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:18.962 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:18.962 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:18.962 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:18.962 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:18.962 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:18.962 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:18.962 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:18.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.959+0000 7f75a764e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:18.972 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.971+0000 7f7597fff640 1 -- 172.21.15.156:0/314097771 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7598001b70 con 0x7f75a01430c0 2024-03-25T18:11:18.972 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.971+0000 7f7597fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:18.973 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.971+0000 7f7597fff640 1 -- 172.21.15.156:0/314097771 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7598001f80 con 0x7f75a01430c0 2024-03-25T18:11:18.973 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.971+0000 7f75a764e640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:18.973 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.971+0000 7f75a764e640 4 set_mon_vals no callback set 2024-03-25T18:11:18.973 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.971+0000 7f7597fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:18.973 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.972+0000 7f7597fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:18.973 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.972+0000 7f7597fff640 10 monclient: dump: 2024-03-25T18:11:18.973 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:18.973 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:18.973 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:18.973 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:18.973 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:18.973 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:18.973 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:18.973 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:18.973 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:18.974 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:18.976 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.972+0000 7f75a764e640 10 monclient: shutdown 2024-03-25T18:11:18.976 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.972+0000 7f75a764e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:18.976 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.972+0000 7f75a764e640 1 -- 172.21.15.156:0/314097771 >> v1:172.21.15.194:6789/0 conn(0x7f75a01430c0 legacy=0x7f75a014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:18.976 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.972+0000 7f75a764e640 1 -- 172.21.15.156:0/314097771 shutdown_connections 2024-03-25T18:11:18.976 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.972+0000 7f75a764e640 1 -- 172.21.15.156:0/314097771 >> 172.21.15.156:0/314097771 conn(0x7f75a00b7ea0 msgr2=0x7f75a00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:18.976 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.972+0000 7f75a764e640 1 -- 172.21.15.156:0/314097771 shutdown_connections 2024-03-25T18:11:18.976 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.972+0000 7f75a764e640 1 -- 172.21.15.156:0/314097771 wait complete. 2024-03-25T18:11:18.976 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.973+0000 7f75a764e640 10 monclient: build_initial_monmap 2024-03-25T18:11:18.976 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.973+0000 7f75a764e640 10 monclient: monmap: 2024-03-25T18:11:18.977 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:11:18.977 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:18.977 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:11:18.977 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:11:18.977 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:18.977 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:11:18.977 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:11:18.977 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:11:18.977 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:11:18.977 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:18.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.974+0000 7f75a764e640 1 librados: starting msgr at 2024-03-25T18:11:18.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.974+0000 7f75a764e640 1 librados: starting objecter 2024-03-25T18:11:18.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.974+0000 7f75a764e640 1 Processor -- start 2024-03-25T18:11:18.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.974+0000 7f75a764e640 1 -- start start 2024-03-25T18:11:18.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.974+0000 7f75a764e640 1 librados: setting wanted keys 2024-03-25T18:11:18.977 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.974+0000 7f75a764e640 1 librados: calling monclient init 2024-03-25T18:11:18.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.974+0000 7f75a764e640 10 monclient: init 2024-03-25T18:11:18.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.974+0000 7f75a764e640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:18.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.974+0000 7f75a764e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:11:18.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.974+0000 7f75a764e640 10 monclient(hunting): picked mon.noname-c con 0x7f75a00a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:11:18.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.975+0000 7f75a764e640 10 monclient(hunting): picked mon.noname-a con 0x7f75a013f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:11:18.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.975+0000 7f75a764e640 10 monclient(hunting): picked mon.noname-b con 0x7f75a01430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:11:18.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.975+0000 7f75a764e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a013c2b0 con 0x7f75a013f710 2024-03-25T18:11:18.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.975+0000 7f75a764e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a0136f10 con 0x7f75a01430c0 2024-03-25T18:11:18.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.975+0000 7f75a764e640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a01370a0 con 0x7f75a00a98f0 2024-03-25T18:11:18.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.975+0000 7f75a764e640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:18.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.975+0000 7f75a764e640 10 monclient(hunting): authenticate will time out at 7160.286058s 2024-03-25T18:11:18.992 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.992+0000 7f75a5beb640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f75a01430c0 0x7f75a0133000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:58106/0 (socket says 172.21.15.156:58106) 2024-03-25T18:11:18.992 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.992+0000 7f75a5beb640 1 -- 172.21.15.156:0/2556323302 learned_addr learned my addr 172.21.15.156:0/2556323302 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:11:18.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.993+0000 7f7595ffb640 10 client.?.objecter ms_handle_connect 0x7f75a01430c0 2024-03-25T18:11:18.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:18.993+0000 7f7595ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:19.001 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.000+0000 7f7595ffb640 10 client.?.objecter ms_handle_connect 0x7f75a00a98f0 2024-03-25T18:11:19.001 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.000+0000 7f7595ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:19.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.006+0000 7f7595ffb640 1 -- 172.21.15.156:0/2556323302 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2423508388 0 0) 0x7f75a0136f10 con 0x7f75a01430c0 2024-03-25T18:11:19.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.006+0000 7f7595ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:11:19.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.006+0000 7f7595ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:19.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.006+0000 7f7595ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:19.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.006+0000 7f7595ffb640 10 monclient(hunting): my global_id is 4357 2024-03-25T18:11:19.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.006+0000 7f7595ffb640 1 -- 172.21.15.156:0/2556323302 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7584004eb0 con 0x7f75a01430c0 2024-03-25T18:11:19.008 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.008+0000 7f7595ffb640 10 client.?.objecter ms_handle_connect 0x7f75a013f710 2024-03-25T18:11:19.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.008+0000 7f7595ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:19.019 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.018+0000 7f7595ffb640 1 -- 172.21.15.156:0/2556323302 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 754388281 0 0) 0x7f75a01370a0 con 0x7f75a00a98f0 2024-03-25T18:11:19.019 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.018+0000 7f7595ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:11:19.019 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.018+0000 7f7595ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:19.019 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.018+0000 7f7595ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:19.019 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.018+0000 7f7595ffb640 10 monclient(hunting): my global_id is 4367 2024-03-25T18:11:19.019 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.018+0000 7f7595ffb640 1 -- 172.21.15.156:0/2556323302 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75a0136f10 con 0x7f75a00a98f0 2024-03-25T18:11:19.022 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.021+0000 7f7595ffb640 1 -- 172.21.15.156:0/2556323302 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3633115623 0 0) 0x7f7584004eb0 con 0x7f75a01430c0 2024-03-25T18:11:19.022 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.021+0000 7f7595ffb640 1 -- 172.21.15.156:0/2556323302 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75a01370a0 con 0x7f75a01430c0 2024-03-25T18:11:19.022 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.021+0000 7f7595ffb640 1 -- 172.21.15.156:0/2556323302 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7598001ec0 con 0x7f75a01430c0 2024-03-25T18:11:19.022 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.021+0000 7f7595ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:19.022 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.021+0000 7f7595ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:19.022 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.021+0000 7f7595ffb640 10 monclient(hunting): dump: 2024-03-25T18:11:19.022 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:19.023 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:19.023 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:19.023 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:19.023 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:19.023 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:19.023 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:19.023 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:19.023 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:19.023 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:19.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.021+0000 7f7595ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:19.029 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.028+0000 7f7595ffb640 1 -- 172.21.15.156:0/2556323302 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 285296058 0 0) 0x7f75a013c2b0 con 0x7f75a013f710 2024-03-25T18:11:19.029 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.028+0000 7f7595ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:11:19.030 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.028+0000 7f7595ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:19.030 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.028+0000 7f7595ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:19.030 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.028+0000 7f7595ffb640 10 monclient(hunting): my global_id is 4359 2024-03-25T18:11:19.030 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.028+0000 7f7595ffb640 1 -- 172.21.15.156:0/2556323302 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7584004eb0 con 0x7f75a013f710 2024-03-25T18:11:19.035 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.034+0000 7f7595ffb640 1 -- 172.21.15.156:0/2556323302 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 574339104 0 0) 0x7f75a01370a0 con 0x7f75a01430c0 2024-03-25T18:11:19.035 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.034+0000 7f7595ffb640 1 -- 172.21.15.156:0/2556323302 >> v1:172.21.15.156:6790/0 conn(0x7f75a00a98f0 legacy=0x7f75a01d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:19.035 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.034+0000 7f7595ffb640 1 -- 172.21.15.156:0/2556323302 >> v1:172.21.15.156:6789/0 conn(0x7f75a013f710 legacy=0x7f75a01377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:19.035 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.034+0000 7f7595ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:11:19.035 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.034+0000 7f7595ffb640 1 monclient: found mon.b 2024-03-25T18:11:19.035 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.034+0000 7f7595ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:19.035 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.034+0000 7f7595ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:11:19.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.034+0000 7f7595ffb640 1 -- 172.21.15.156:0/2556323302 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a0137230 con 0x7f75a01430c0 2024-03-25T18:11:19.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.034+0000 7f7595ffb640 10 monclient: _finish_auth 0 2024-03-25T18:11:19.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.034+0000 7f7595ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:11:19.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.034+0000 7f7595ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:19.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.034+0000 7f75a764e640 5 monclient: authenticate success, global_id 4357 2024-03-25T18:11:19.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.034+0000 7f75a764e640 10 monclient: _renew_subs 2024-03-25T18:11:19.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.034+0000 7f75a764e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:11:19.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.034+0000 7f75a764e640 1 -- 172.21.15.156:0/2556323302 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a013b620 con 0x7f75a01430c0 2024-03-25T18:11:19.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.034+0000 7f75a764e640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:19.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.034+0000 7f75a764e640 10 monclient: _renew_subs 2024-03-25T18:11:19.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.034+0000 7f75a764e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:11:19.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.034+0000 7f75a764e640 1 -- 172.21.15.156:0/2556323302 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75a0133710 con 0x7f75a01430c0 2024-03-25T18:11:19.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.034+0000 7f75a764e640 1 librados: init done 2024-03-25T18:11:19.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.035+0000 7f75a764e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:19.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.035+0000 7f75a764e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:19.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.035+0000 7f75a764e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:11:19.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.035+0000 7f75a764e640 1 -- 172.21.15.156:0/2556323302 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75a00627b0 con 0x7f75a01430c0 2024-03-25T18:11:19.312 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.311+0000 7f7595ffb640 1 -- 172.21.15.156:0/2556323302 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7598004e10 con 0x7f75a01430c0 2024-03-25T18:11:19.312 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.311+0000 7f7595ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:19.312 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.311+0000 7f7595ffb640 1 -- 172.21.15.156:0/2556323302 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7598002100 con 0x7f75a01430c0 2024-03-25T18:11:19.312 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.311+0000 7f7595ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:19.313 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.311+0000 7f7595ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:19.313 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.311+0000 7f7595ffb640 10 monclient: dump: 2024-03-25T18:11:19.313 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:19.313 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:19.313 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:19.313 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:19.313 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:19.313 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:19.313 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:19.313 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:19.313 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:19.313 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:19.313 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.311+0000 7f75977fe640 4 set_mon_vals no callback set 2024-03-25T18:11:19.314 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.312+0000 7f7595ffb640 1 -- 172.21.15.156:0/2556323302 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f759801adf0 con 0x7f75a01430c0 2024-03-25T18:11:19.314 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.312+0000 7f75a53ea640 10 monclient: get_auth_request con 0x7f758406de00 auth_method 0 2024-03-25T18:11:19.314 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.313+0000 7f7595ffb640 10 client.4357.objecter ms_handle_connect 0x7f758406de00 2024-03-25T18:11:19.316 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.315+0000 7f7595ffb640 1 -- 172.21.15.156:0/2556323302 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f7598054970 con 0x7f75a01430c0 2024-03-25T18:11:19.316 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.315+0000 7f7595ffb640 10 client.4357.objecter ms_dispatch 0x7f75a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:19.316 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.315+0000 7f7595ffb640 3 client.4357.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:19.316 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.315+0000 7f7595ffb640 3 client.4357.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:19.316 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.315+0000 7f7595ffb640 20 client.4357.objecter dump_active .. 0 homeless 2024-03-25T18:11:19.318 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.317+0000 7f7595ffb640 1 -- 172.21.15.156:0/2556323302 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7598050480 con 0x7f75a01430c0 2024-03-25T18:11:19.318 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.317+0000 7f7595ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:19.318 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.317+0000 7f7595ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:19.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.553+0000 7f75a764e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:19.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.553+0000 7f75a764e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:19.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.553+0000 7f75a764e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:11:19.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.553+0000 7f75a764e640 1 -- 172.21.15.156:0/2556323302 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75a0133a60 con 0x7f75a01430c0 2024-03-25T18:11:19.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.589+0000 7f7595ffb640 1 -- 172.21.15.156:0/2556323302 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f75980523b0 con 0x7f75a01430c0 2024-03-25T18:11:19.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.589+0000 7f7595ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:19.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.589+0000 7f7595ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:19.591 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:11:19.591 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:19.593 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.591+0000 7f75a764e640 10 librados: watch_flush enter 2024-03-25T18:11:19.593 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.591+0000 7f75a764e640 10 librados: watch_flush exit 2024-03-25T18:11:19.593 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.591+0000 7f75a764e640 20 client.4357.objecter shutdown clearing up homeless session... 2024-03-25T18:11:19.593 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.591+0000 7f75a764e640 10 client.4357.objecter successfully canceled tick 2024-03-25T18:11:19.593 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.592+0000 7f75a764e640 1 -- 172.21.15.156:0/2556323302 >> v1:172.21.15.194:6800/2946427713 conn(0x7f758406de00 legacy=0x7f75840702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:19.593 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.592+0000 7f75a764e640 10 monclient: shutdown 2024-03-25T18:11:19.593 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.592+0000 7f75a764e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:19.593 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.592+0000 7f75a764e640 1 -- 172.21.15.156:0/2556323302 >> v1:172.21.15.194:6789/0 conn(0x7f75a01430c0 legacy=0x7f75a0133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:19.593 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.592+0000 7f75a764e640 1 -- 172.21.15.156:0/2556323302 shutdown_connections 2024-03-25T18:11:19.593 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.592+0000 7f75a764e640 1 -- 172.21.15.156:0/2556323302 >> 172.21.15.156:0/2556323302 conn(0x7f75a00b7ea0 msgr2=0x7f75a0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:19.594 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.592+0000 7f75a764e640 1 -- 172.21.15.156:0/2556323302 shutdown_connections 2024-03-25T18:11:19.594 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.592+0000 7f75a764e640 1 -- 172.21.15.156:0/2556323302 wait complete. 2024-03-25T18:11:19.594 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:19.592+0000 7f75a764e640 1 librados: shutdown 2024-03-25T18:11:19.606 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:23.658 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:03:29.649 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:23.664 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:03:29.654 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:23.690 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:03:29.680 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:23.724 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:03:29.691 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:23.731 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:03:29.698 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:23.747 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:03:29.738 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:23.766 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:03:29.733 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:23.766 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:03:29.756 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:23.794 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:03:29.785 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:26.413 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:26.529 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.528+0000 7f7e38b77640 10 monclient: get_monmap_and_config 2024-03-25T18:11:26.529 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.528+0000 7f7e38b77640 10 monclient: build_initial_monmap 2024-03-25T18:11:26.529 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.528+0000 7f7e38b77640 10 monclient: monmap: 2024-03-25T18:11:26.529 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:11:26.529 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:26.529 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:11:26.529147+0000 2024-03-25T18:11:26.529 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:11:26.529147+0000 2024-03-25T18:11:26.530 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:26.530 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:26.530 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:11:26.530 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:11:26.530 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:11:26.530 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:26.531 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.529+0000 7f7e38b77640 1 Processor -- start 2024-03-25T18:11:26.531 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.529+0000 7f7e38b77640 1 -- start start 2024-03-25T18:11:26.531 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.529+0000 7f7e38b77640 10 monclient: init 2024-03-25T18:11:26.531 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.529+0000 7f7e38b77640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:26.531 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.529+0000 7f7e38b77640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:11:26.531 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.529+0000 7f7e38b77640 10 monclient(hunting): picked mon.noname-a con 0x7f7e340a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:11:26.531 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.529+0000 7f7e38b77640 10 monclient(hunting): picked mon.noname-b con 0x7f7e3413f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:11:26.532 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.529+0000 7f7e38b77640 10 monclient(hunting): picked mon.noname-c con 0x7f7e341430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:11:26.532 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.529+0000 7f7e38b77640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e34142520 con 0x7f7e340a98f0 2024-03-25T18:11:26.532 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.530+0000 7f7e38b77640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e340b28a0 con 0x7f7e341430c0 2024-03-25T18:11:26.532 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.530+0000 7f7e38b77640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e340b2a30 con 0x7f7e3413f710 2024-03-25T18:11:26.532 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.530+0000 7f7e38b77640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:26.532 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.530+0000 7f7e38b77640 10 monclient(hunting): authenticate will time out at 7167.840816s 2024-03-25T18:11:26.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.534+0000 7f7e2bfff640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f7e3413f710 0x7f7e341429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:42198/0 (socket says 172.21.15.156:42198) 2024-03-25T18:11:26.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.534+0000 7f7e2bfff640 1 -- 172.21.15.156:0/1112940965 learned_addr learned my addr 172.21.15.156:0/1112940965 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:11:26.541 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.541+0000 7f7e31d9b640 1 -- 172.21.15.156:0/1112940965 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181954853 0 0) 0x7f7e34142520 con 0x7f7e340a98f0 2024-03-25T18:11:26.542 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.541+0000 7f7e31d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:11:26.542 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.541+0000 7f7e31d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:26.542 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.541+0000 7f7e31d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:26.542 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.541+0000 7f7e31d9b640 10 monclient(hunting): my global_id is 4362 2024-03-25T18:11:26.542 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.541+0000 7f7e31d9b640 1 -- 172.21.15.156:0/1112940965 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e20003520 con 0x7f7e340a98f0 2024-03-25T18:11:26.551 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.551+0000 7f7e31d9b640 1 -- 172.21.15.156:0/1112940965 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3165462848 0 0) 0x7f7e340b28a0 con 0x7f7e341430c0 2024-03-25T18:11:26.552 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.551+0000 7f7e31d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:11:26.552 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.551+0000 7f7e31d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:26.552 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.551+0000 7f7e31d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:26.552 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.551+0000 7f7e31d9b640 10 monclient(hunting): my global_id is 4360 2024-03-25T18:11:26.552 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.551+0000 7f7e31d9b640 1 -- 172.21.15.156:0/1112940965 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e34142520 con 0x7f7e341430c0 2024-03-25T18:11:26.553 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.553+0000 7f7e31d9b640 1 -- 172.21.15.156:0/1112940965 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2897823378 0 0) 0x7f7e340b2a30 con 0x7f7e3413f710 2024-03-25T18:11:26.553 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.553+0000 7f7e31d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:11:26.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.553+0000 7f7e31d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:26.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.553+0000 7f7e31d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:26.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.553+0000 7f7e31d9b640 10 monclient(hunting): my global_id is 4370 2024-03-25T18:11:26.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.553+0000 7f7e31d9b640 1 -- 172.21.15.156:0/1112940965 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e340b28a0 con 0x7f7e3413f710 2024-03-25T18:11:26.560 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.558+0000 7f7e31d9b640 1 -- 172.21.15.156:0/1112940965 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2036854069 0 0) 0x7f7e34142520 con 0x7f7e341430c0 2024-03-25T18:11:26.560 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.558+0000 7f7e31d9b640 1 -- 172.21.15.156:0/1112940965 >> v1:172.21.15.156:6790/0 conn(0x7f7e3413f710 legacy=0x7f7e341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:26.560 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.558+0000 7f7e31d9b640 1 -- 172.21.15.156:0/1112940965 >> v1:172.21.15.156:6789/0 conn(0x7f7e340a98f0 legacy=0x7f7e340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:26.560 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.558+0000 7f7e31d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:11:26.560 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.559+0000 7f7e31d9b640 1 monclient: found mon.noname-c 2024-03-25T18:11:26.560 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.559+0000 7f7e31d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:26.560 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.559+0000 7f7e31d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:11:26.560 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.559+0000 7f7e31d9b640 1 -- 172.21.15.156:0/1112940965 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e340b44c0 con 0x7f7e341430c0 2024-03-25T18:11:26.560 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.559+0000 7f7e31d9b640 10 monclient: _finish_auth 0 2024-03-25T18:11:26.560 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.559+0000 7f7e31d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:11:26.561 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.559+0000 7f7e31d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:26.561 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.559+0000 7f7e31d9b640 1 -- 172.21.15.156:0/1112940965 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7e2c0019f0 con 0x7f7e341430c0 2024-03-25T18:11:26.561 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.559+0000 7f7e31d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:26.561 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.559+0000 7f7e31d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:26.561 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.559+0000 7f7e31d9b640 10 monclient: dump: 2024-03-25T18:11:26.561 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:26.561 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:26.561 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:26.561 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:26.561 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:26.561 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:26.561 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:26.561 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:26.561 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:26.561 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:26.562 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.559+0000 7f7e38b77640 5 monclient: authenticate success, global_id 4360 2024-03-25T18:11:26.562 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.559+0000 7f7e38b77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:26.563 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.563+0000 7f7e31d9b640 1 -- 172.21.15.156:0/1112940965 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e2c001b50 con 0x7f7e341430c0 2024-03-25T18:11:26.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.563+0000 7f7e31d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:26.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.563+0000 7f7e31d9b640 1 -- 172.21.15.156:0/1112940965 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7e2c001f60 con 0x7f7e341430c0 2024-03-25T18:11:26.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.563+0000 7f7e38b77640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:26.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.563+0000 7f7e38b77640 4 set_mon_vals no callback set 2024-03-25T18:11:26.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.563+0000 7f7e31d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:26.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.563+0000 7f7e31d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:26.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.563+0000 7f7e31d9b640 10 monclient: dump: 2024-03-25T18:11:26.564 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:26.564 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:26.564 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:26.564 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:26.564 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:26.564 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:26.564 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:26.564 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:26.565 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:26.565 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:26.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.564+0000 7f7e38b77640 10 monclient: shutdown 2024-03-25T18:11:26.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.564+0000 7f7e38b77640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:26.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.564+0000 7f7e38b77640 1 -- 172.21.15.156:0/1112940965 >> v1:172.21.15.194:6789/0 conn(0x7f7e341430c0 legacy=0x7f7e3414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:26.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.564+0000 7f7e38b77640 1 -- 172.21.15.156:0/1112940965 shutdown_connections 2024-03-25T18:11:26.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.564+0000 7f7e38b77640 1 -- 172.21.15.156:0/1112940965 >> 172.21.15.156:0/1112940965 conn(0x7f7e340b7ea0 msgr2=0x7f7e340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:26.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.564+0000 7f7e38b77640 1 -- 172.21.15.156:0/1112940965 shutdown_connections 2024-03-25T18:11:26.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.564+0000 7f7e38b77640 1 -- 172.21.15.156:0/1112940965 wait complete. 2024-03-25T18:11:26.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.565+0000 7f7e38b77640 10 monclient: build_initial_monmap 2024-03-25T18:11:26.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.565+0000 7f7e38b77640 10 monclient: monmap: 2024-03-25T18:11:26.567 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:11:26.567 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:26.567 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:11:26.567 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:11:26.567 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:26.567 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:11:26.567 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:11:26.567 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:11:26.567 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:11:26.567 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:26.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.565+0000 7f7e38b77640 1 librados: starting msgr at 2024-03-25T18:11:26.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.565+0000 7f7e38b77640 1 librados: starting objecter 2024-03-25T18:11:26.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.565+0000 7f7e38b77640 1 Processor -- start 2024-03-25T18:11:26.568 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.565+0000 7f7e38b77640 1 -- start start 2024-03-25T18:11:26.568 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.565+0000 7f7e38b77640 1 librados: setting wanted keys 2024-03-25T18:11:26.568 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.565+0000 7f7e38b77640 1 librados: calling monclient init 2024-03-25T18:11:26.568 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.565+0000 7f7e38b77640 10 monclient: init 2024-03-25T18:11:26.569 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.566+0000 7f7e38b77640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:26.569 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.566+0000 7f7e38b77640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:11:26.569 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.566+0000 7f7e38b77640 10 monclient(hunting): picked mon.noname-c con 0x7f7e340a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:11:26.569 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.566+0000 7f7e38b77640 10 monclient(hunting): picked mon.noname-a con 0x7f7e3413f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:11:26.569 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.566+0000 7f7e38b77640 10 monclient(hunting): picked mon.noname-b con 0x7f7e341430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:11:26.569 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.566+0000 7f7e38b77640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e34137ce0 con 0x7f7e3413f710 2024-03-25T18:11:26.569 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.566+0000 7f7e38b77640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e341396e0 con 0x7f7e341430c0 2024-03-25T18:11:26.569 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.566+0000 7f7e38b77640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e341399c0 con 0x7f7e340a98f0 2024-03-25T18:11:26.569 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.566+0000 7f7e38b77640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:26.569 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.566+0000 7f7e38b77640 10 monclient(hunting): authenticate will time out at 7167.877722s 2024-03-25T18:11:26.584 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.583+0000 7f7e32d9d640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f7e341430c0 0x7f7e341375d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:53734/0 (socket says 172.21.15.156:53734) 2024-03-25T18:11:26.584 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.583+0000 7f7e2bfff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f7e3413f710 0x7f7e34138320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:39558/0 (socket says 172.21.15.156:39558) 2024-03-25T18:11:26.585 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.583+0000 7f7e32d9d640 1 -- 172.21.15.156:0/2429271419 learned_addr learned my addr 172.21.15.156:0/2429271419 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:11:26.587 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.586+0000 7f7e2affd640 10 client.?.objecter ms_handle_connect 0x7f7e341430c0 2024-03-25T18:11:26.587 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.586+0000 7f7e2affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:26.587 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.586+0000 7f7e2affd640 10 client.?.objecter ms_handle_connect 0x7f7e3413f710 2024-03-25T18:11:26.587 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.586+0000 7f7e2affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:26.592 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.591+0000 7f7e2affd640 10 client.?.objecter ms_handle_connect 0x7f7e340a98f0 2024-03-25T18:11:26.592 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.591+0000 7f7e2affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:26.593 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.592+0000 7f7e2affd640 1 -- 172.21.15.156:0/2429271419 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1166847351 0 0) 0x7f7e34137ce0 con 0x7f7e3413f710 2024-03-25T18:11:26.593 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.592+0000 7f7e2affd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:26.593 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.592+0000 7f7e2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:26.593 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.592+0000 7f7e2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:26.593 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.592+0000 7f7e2affd640 10 monclient(hunting): my global_id is 4365 2024-03-25T18:11:26.593 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.592+0000 7f7e2affd640 1 -- 172.21.15.156:0/2429271419 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e14004eb0 con 0x7f7e3413f710 2024-03-25T18:11:26.601 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.601+0000 7f7e2affd640 1 -- 172.21.15.156:0/2429271419 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2243540532 0 0) 0x7f7e341399c0 con 0x7f7e340a98f0 2024-03-25T18:11:26.601 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.601+0000 7f7e2affd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:26.601 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.601+0000 7f7e2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:26.602 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.601+0000 7f7e2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:26.602 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.601+0000 7f7e2affd640 10 monclient(hunting): my global_id is 4373 2024-03-25T18:11:26.602 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.601+0000 7f7e2affd640 1 -- 172.21.15.156:0/2429271419 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e34137ce0 con 0x7f7e340a98f0 2024-03-25T18:11:26.605 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.605+0000 7f7e2affd640 1 -- 172.21.15.156:0/2429271419 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085509759 0 0) 0x7f7e341396e0 con 0x7f7e341430c0 2024-03-25T18:11:26.606 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.605+0000 7f7e2affd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:26.606 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.605+0000 7f7e2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:26.606 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.605+0000 7f7e2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:26.606 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.605+0000 7f7e2affd640 10 monclient(hunting): my global_id is 4363 2024-03-25T18:11:26.606 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.605+0000 7f7e2affd640 1 -- 172.21.15.156:0/2429271419 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e341399c0 con 0x7f7e341430c0 2024-03-25T18:11:26.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.608+0000 7f7e2affd640 1 -- 172.21.15.156:0/2429271419 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 506634957 0 0) 0x7f7e14004eb0 con 0x7f7e3413f710 2024-03-25T18:11:26.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.608+0000 7f7e2affd640 1 -- 172.21.15.156:0/2429271419 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e341396e0 con 0x7f7e3413f710 2024-03-25T18:11:26.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.608+0000 7f7e2affd640 1 -- 172.21.15.156:0/2429271419 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7e1c0018f0 con 0x7f7e3413f710 2024-03-25T18:11:26.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.608+0000 7f7e2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:26.610 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.608+0000 7f7e2affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:26.610 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.608+0000 7f7e2affd640 10 monclient(hunting): dump: 2024-03-25T18:11:26.610 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:26.610 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:26.610 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:26.610 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:26.610 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:26.610 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:26.610 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:26.610 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:26.610 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:26.610 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:26.610 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.608+0000 7f7e2affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:26.616 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.615+0000 7f7e2affd640 1 -- 172.21.15.156:0/2429271419 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 97418062 0 0) 0x7f7e341399c0 con 0x7f7e341430c0 2024-03-25T18:11:26.616 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.615+0000 7f7e2affd640 1 -- 172.21.15.156:0/2429271419 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e14004eb0 con 0x7f7e341430c0 2024-03-25T18:11:26.616 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.615+0000 7f7e2affd640 1 -- 172.21.15.156:0/2429271419 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7e2c001f20 con 0x7f7e341430c0 2024-03-25T18:11:26.616 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.615+0000 7f7e2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:26.616 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.615+0000 7f7e2affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:26.616 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.615+0000 7f7e2affd640 10 monclient(hunting): dump: 2024-03-25T18:11:26.616 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:26.616 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:26.616 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:26.616 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:26.617 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:26.617 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:26.617 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:26.617 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:26.617 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:26.617 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:26.621 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.619+0000 7f7e2affd640 1 -- 172.21.15.156:0/2429271419 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1218450669 0 0) 0x7f7e341396e0 con 0x7f7e3413f710 2024-03-25T18:11:26.621 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.619+0000 7f7e2affd640 1 -- 172.21.15.156:0/2429271419 >> v1:172.21.15.156:6790/0 conn(0x7f7e340a98f0 legacy=0x7f7e341da780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:26.621 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.619+0000 7f7e2affd640 1 -- 172.21.15.156:0/2429271419 >> v1:172.21.15.194:6789/0 conn(0x7f7e341430c0 legacy=0x7f7e341375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:26.621 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.619+0000 7f7e2affd640 10 monclient: _finish_hunting 0 2024-03-25T18:11:26.621 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.619+0000 7f7e2affd640 1 monclient: found mon.a 2024-03-25T18:11:26.621 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.619+0000 7f7e2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:26.621 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.619+0000 7f7e2affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:11:26.621 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.619+0000 7f7e2affd640 1 -- 172.21.15.156:0/2429271419 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e340b0fb0 con 0x7f7e3413f710 2024-03-25T18:11:26.621 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.619+0000 7f7e2affd640 10 monclient: _finish_auth 0 2024-03-25T18:11:26.621 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.619+0000 7f7e2affd640 10 monclient: _check_auth_tickets 2024-03-25T18:11:26.622 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.619+0000 7f7e2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:26.622 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.620+0000 7f7e38b77640 5 monclient: authenticate success, global_id 4365 2024-03-25T18:11:26.622 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.620+0000 7f7e38b77640 10 monclient: _renew_subs 2024-03-25T18:11:26.622 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.620+0000 7f7e38b77640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:11:26.622 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.620+0000 7f7e38b77640 1 -- 172.21.15.156:0/2429271419 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e340afdc0 con 0x7f7e3413f710 2024-03-25T18:11:26.622 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.620+0000 7f7e38b77640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:26.622 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.620+0000 7f7e38b77640 10 monclient: _renew_subs 2024-03-25T18:11:26.622 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.620+0000 7f7e38b77640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:11:26.622 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.620+0000 7f7e38b77640 1 -- 172.21.15.156:0/2429271419 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e340b02a0 con 0x7f7e3413f710 2024-03-25T18:11:26.622 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.620+0000 7f7e38b77640 1 librados: init done 2024-03-25T18:11:26.623 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.621+0000 7f7e38b77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:26.623 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.621+0000 7f7e38b77640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:26.623 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.621+0000 7f7e38b77640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:11:26.623 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.621+0000 7f7e38b77640 1 -- 172.21.15.156:0/2429271419 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e340627b0 con 0x7f7e3413f710 2024-03-25T18:11:26.837 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.837+0000 7f7e2affd640 1 -- 172.21.15.156:0/2429271419 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e1c001d40 con 0x7f7e3413f710 2024-03-25T18:11:26.838 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.837+0000 7f7e2affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:26.838 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.837+0000 7f7e2affd640 1 -- 172.21.15.156:0/2429271419 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7e1c004a00 con 0x7f7e3413f710 2024-03-25T18:11:26.838 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.837+0000 7f7e2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:26.838 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.837+0000 7f7e2affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:26.838 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.837+0000 7f7e2affd640 10 monclient: dump: 2024-03-25T18:11:26.838 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:26.838 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:26.838 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:26.838 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:26.838 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:26.838 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:26.838 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:26.838 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:26.839 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:26.839 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:26.839 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.837+0000 7f7e3159a640 4 set_mon_vals no callback set 2024-03-25T18:11:26.918 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.918+0000 7f7e2affd640 1 -- 172.21.15.156:0/2429271419 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f7e1c004be0 con 0x7f7e3413f710 2024-03-25T18:11:26.919 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.918+0000 7f7e3259c640 10 monclient: get_auth_request con 0x7f7e1406df70 auth_method 0 2024-03-25T18:11:26.919 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.919+0000 7f7e2affd640 10 client.4365.objecter ms_handle_connect 0x7f7e1406df70 2024-03-25T18:11:26.924 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.923+0000 7f7e2affd640 1 -- 172.21.15.156:0/2429271419 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f7e1c054850 con 0x7f7e3413f710 2024-03-25T18:11:26.924 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.923+0000 7f7e2affd640 10 client.4365.objecter ms_dispatch 0x7f7e34000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:26.924 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.923+0000 7f7e2affd640 3 client.4365.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:26.924 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.923+0000 7f7e2affd640 3 client.4365.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:26.924 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.923+0000 7f7e2affd640 20 client.4365.objecter dump_active .. 0 homeless 2024-03-25T18:11:26.924 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.923+0000 7f7e2affd640 1 -- 172.21.15.156:0/2429271419 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7e1c053f20 con 0x7f7e3413f710 2024-03-25T18:11:26.924 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.923+0000 7f7e2affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:26.924 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:26.923+0000 7f7e2affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:27.170 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:27.169+0000 7f7e38b77640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:27.170 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:27.169+0000 7f7e38b77640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:27.171 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:27.169+0000 7f7e38b77640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:11:27.171 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:27.169+0000 7f7e38b77640 1 -- 172.21.15.156:0/2429271419 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e340b71a0 con 0x7f7e3413f710 2024-03-25T18:11:27.197 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:27.196+0000 7f7e2affd640 1 -- 172.21.15.156:0/2429271419 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f7e1c052290 con 0x7f7e3413f710 2024-03-25T18:11:27.197 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:27.196+0000 7f7e2affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:27.197 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:27.196+0000 7f7e2affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:27.197 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:11:27.197 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:27.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:27.198+0000 7f7e38b77640 10 librados: watch_flush enter 2024-03-25T18:11:27.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:27.199+0000 7f7e38b77640 10 librados: watch_flush exit 2024-03-25T18:11:27.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:27.199+0000 7f7e38b77640 20 client.4365.objecter shutdown clearing up homeless session... 2024-03-25T18:11:27.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:27.199+0000 7f7e38b77640 10 client.4365.objecter successfully canceled tick 2024-03-25T18:11:27.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:27.199+0000 7f7e38b77640 1 -- 172.21.15.156:0/2429271419 >> v1:172.21.15.194:6800/2946427713 conn(0x7f7e1406df70 legacy=0x7f7e14070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:27.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:27.199+0000 7f7e38b77640 10 monclient: shutdown 2024-03-25T18:11:27.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:27.199+0000 7f7e38b77640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:27.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:27.199+0000 7f7e38b77640 1 -- 172.21.15.156:0/2429271419 >> v1:172.21.15.156:6789/0 conn(0x7f7e3413f710 legacy=0x7f7e34138320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:27.201 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:27.199+0000 7f7e38b77640 1 -- 172.21.15.156:0/2429271419 shutdown_connections 2024-03-25T18:11:27.201 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:27.199+0000 7f7e38b77640 1 -- 172.21.15.156:0/2429271419 >> 172.21.15.156:0/2429271419 conn(0x7f7e340b7ea0 msgr2=0x7f7e34074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:27.201 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:27.199+0000 7f7e38b77640 1 -- 172.21.15.156:0/2429271419 shutdown_connections 2024-03-25T18:11:27.201 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:27.199+0000 7f7e38b77640 1 -- 172.21.15.156:0/2429271419 wait complete. 2024-03-25T18:11:27.201 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:27.199+0000 7f7e38b77640 1 librados: shutdown 2024-03-25T18:11:27.211 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:34.018 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:34.135 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.134+0000 7ff8b2c72640 10 monclient: get_monmap_and_config 2024-03-25T18:11:34.135 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.134+0000 7ff8b2c72640 10 monclient: build_initial_monmap 2024-03-25T18:11:34.135 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.134+0000 7ff8b2c72640 10 monclient: monmap: 2024-03-25T18:11:34.135 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:11:34.135 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:34.135 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:11:34.135354+0000 2024-03-25T18:11:34.135 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:11:34.135354+0000 2024-03-25T18:11:34.136 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:34.136 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:34.136 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:11:34.136 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:11:34.136 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:11:34.136 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:34.137 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.135+0000 7ff8b2c72640 1 Processor -- start 2024-03-25T18:11:34.137 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.135+0000 7ff8b2c72640 1 -- start start 2024-03-25T18:11:34.137 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.135+0000 7ff8b2c72640 10 monclient: init 2024-03-25T18:11:34.137 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.136+0000 7ff8b2c72640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:34.137 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.136+0000 7ff8b2c72640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:11:34.137 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.136+0000 7ff8b2c72640 10 monclient(hunting): picked mon.noname-a con 0x7ff8ac0a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:11:34.137 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.136+0000 7ff8b2c72640 10 monclient(hunting): picked mon.noname-c con 0x7ff8ac13f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:11:34.137 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.136+0000 7ff8b2c72640 10 monclient(hunting): picked mon.noname-b con 0x7ff8ac1430c0 addr v1:172.21.15.156:6790/0 2024-03-25T18:11:34.137 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.136+0000 7ff8b2c72640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ac142520 con 0x7ff8ac0a98f0 2024-03-25T18:11:34.137 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.136+0000 7ff8b2c72640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ac0b28a0 con 0x7ff8ac13f710 2024-03-25T18:11:34.138 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.136+0000 7ff8b2c72640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ac0b2a30 con 0x7ff8ac1430c0 2024-03-25T18:11:34.138 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.136+0000 7ff8b2c72640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:34.138 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.136+0000 7ff8b2c72640 10 monclient(hunting): authenticate will time out at 7175.447007s 2024-03-25T18:11:34.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.145+0000 7ff8abfff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7ff8ac13f710 0x7ff8ac1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:35260/0 (socket says 172.21.15.156:35260) 2024-03-25T18:11:34.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.145+0000 7ff8abfff640 1 -- 172.21.15.156:0/4084232447 learned_addr learned my addr 172.21.15.156:0/4084232447 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:11:34.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.161+0000 7ff8ab7fe640 1 -- 172.21.15.156:0/4084232447 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415213197 0 0) 0x7ff8ac142520 con 0x7ff8ac0a98f0 2024-03-25T18:11:34.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.161+0000 7ff8ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:34.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.161+0000 7ff8ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:34.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.161+0000 7ff8ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:34.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.161+0000 7ff8ab7fe640 10 monclient(hunting): my global_id is 4368 2024-03-25T18:11:34.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.161+0000 7ff8ab7fe640 1 -- 172.21.15.156:0/4084232447 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff89c003520 con 0x7ff8ac0a98f0 2024-03-25T18:11:34.165 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.164+0000 7ff8ab7fe640 1 -- 172.21.15.156:0/4084232447 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927320258 0 0) 0x7ff8ac0b28a0 con 0x7ff8ac13f710 2024-03-25T18:11:34.165 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.164+0000 7ff8ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:34.165 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.164+0000 7ff8ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:34.165 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.164+0000 7ff8ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:34.165 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.164+0000 7ff8ab7fe640 10 monclient(hunting): my global_id is 4366 2024-03-25T18:11:34.165 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.164+0000 7ff8ab7fe640 1 -- 172.21.15.156:0/4084232447 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8ac142520 con 0x7ff8ac13f710 2024-03-25T18:11:34.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.168+0000 7ff8ab7fe640 1 -- 172.21.15.156:0/4084232447 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4251917984 0 0) 0x7ff89c003520 con 0x7ff8ac0a98f0 2024-03-25T18:11:34.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.168+0000 7ff8ab7fe640 1 -- 172.21.15.156:0/4084232447 >> v1:172.21.15.156:6790/0 conn(0x7ff8ac1430c0 legacy=0x7ff8ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:34.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.168+0000 7ff8ab7fe640 1 -- 172.21.15.156:0/4084232447 >> v1:172.21.15.194:6789/0 conn(0x7ff8ac13f710 legacy=0x7ff8ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:34.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.168+0000 7ff8ab7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:11:34.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.168+0000 7ff8ab7fe640 1 monclient: found mon.noname-a 2024-03-25T18:11:34.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.168+0000 7ff8ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:34.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.168+0000 7ff8ab7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:11:34.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.168+0000 7ff8ab7fe640 1 -- 172.21.15.156:0/4084232447 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ac0b44c0 con 0x7ff8ac0a98f0 2024-03-25T18:11:34.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.168+0000 7ff8ab7fe640 10 monclient: _finish_auth 0 2024-03-25T18:11:34.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.168+0000 7ff8ab7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:11:34.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.168+0000 7ff8ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:34.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.168+0000 7ff8ab7fe640 1 -- 172.21.15.156:0/4084232447 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7ff8a0001690 con 0x7ff8ac0a98f0 2024-03-25T18:11:34.170 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.168+0000 7ff8b2c72640 5 monclient: authenticate success, global_id 4368 2024-03-25T18:11:34.170 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.168+0000 7ff8b2c72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:34.170 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.168+0000 7ff8ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:34.170 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.168+0000 7ff8ab7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:34.170 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.168+0000 7ff8ab7fe640 10 monclient: dump: 2024-03-25T18:11:34.170 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:34.170 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:34.170 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:34.170 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:34.170 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:34.170 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:34.170 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:34.170 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:34.171 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:34.171 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:34.171 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.168+0000 7ff8b2c72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:34.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.172+0000 7ff8ab7fe640 1 -- 172.21.15.156:0/4084232447 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8a00017f0 con 0x7ff8ac0a98f0 2024-03-25T18:11:34.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.172+0000 7ff8ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:34.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.172+0000 7ff8ab7fe640 1 -- 172.21.15.156:0/4084232447 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7ff8a0001c00 con 0x7ff8ac0a98f0 2024-03-25T18:11:34.173 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.172+0000 7ff8b2c72640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:34.173 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.172+0000 7ff8b2c72640 4 set_mon_vals no callback set 2024-03-25T18:11:34.173 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.172+0000 7ff8ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:34.173 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.172+0000 7ff8ab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:34.173 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.172+0000 7ff8ab7fe640 10 monclient: dump: 2024-03-25T18:11:34.173 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:34.173 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:34.173 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:34.173 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:34.173 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:34.173 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:34.173 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:34.173 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:34.174 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:34.174 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:34.175 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.172+0000 7ff8b2c72640 10 monclient: shutdown 2024-03-25T18:11:34.175 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.172+0000 7ff8b2c72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:34.175 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.172+0000 7ff8b2c72640 1 -- 172.21.15.156:0/4084232447 >> v1:172.21.15.156:6789/0 conn(0x7ff8ac0a98f0 legacy=0x7ff8ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:34.175 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.172+0000 7ff8b2c72640 1 -- 172.21.15.156:0/4084232447 shutdown_connections 2024-03-25T18:11:34.175 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.173+0000 7ff8b2c72640 1 -- 172.21.15.156:0/4084232447 >> 172.21.15.156:0/4084232447 conn(0x7ff8ac0b7ea0 msgr2=0x7ff8ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:34.175 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.173+0000 7ff8b2c72640 1 -- 172.21.15.156:0/4084232447 shutdown_connections 2024-03-25T18:11:34.175 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.173+0000 7ff8b2c72640 1 -- 172.21.15.156:0/4084232447 wait complete. 2024-03-25T18:11:34.175 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.173+0000 7ff8b2c72640 10 monclient: build_initial_monmap 2024-03-25T18:11:34.176 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.173+0000 7ff8b2c72640 10 monclient: monmap: 2024-03-25T18:11:34.176 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:11:34.176 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:34.176 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:11:34.176 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:11:34.176 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:34.176 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:11:34.176 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:11:34.176 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:11:34.176 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:11:34.176 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:34.176 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.174+0000 7ff8b2c72640 1 librados: starting msgr at 2024-03-25T18:11:34.176 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.174+0000 7ff8b2c72640 1 librados: starting objecter 2024-03-25T18:11:34.176 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.174+0000 7ff8b2c72640 1 Processor -- start 2024-03-25T18:11:34.176 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.174+0000 7ff8b2c72640 1 -- start start 2024-03-25T18:11:34.176 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.174+0000 7ff8b2c72640 1 librados: setting wanted keys 2024-03-25T18:11:34.176 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.174+0000 7ff8b2c72640 1 librados: calling monclient init 2024-03-25T18:11:34.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.174+0000 7ff8b2c72640 10 monclient: init 2024-03-25T18:11:34.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.175+0000 7ff8b2c72640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:34.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.175+0000 7ff8b2c72640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:11:34.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.175+0000 7ff8b2c72640 10 monclient(hunting): picked mon.noname-c con 0x7ff8ac0a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:11:34.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.175+0000 7ff8b2c72640 10 monclient(hunting): picked mon.noname-b con 0x7ff8ac13f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:11:34.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.175+0000 7ff8b2c72640 10 monclient(hunting): picked mon.noname-a con 0x7ff8ac1430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:11:34.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.175+0000 7ff8b2c72640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ac132a30 con 0x7ff8ac1430c0 2024-03-25T18:11:34.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.175+0000 7ff8b2c72640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ac138c80 con 0x7ff8ac13f710 2024-03-25T18:11:34.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.175+0000 7ff8b2c72640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ac138f60 con 0x7ff8ac0a98f0 2024-03-25T18:11:34.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.175+0000 7ff8b2c72640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:34.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.175+0000 7ff8b2c72640 10 monclient(hunting): authenticate will time out at 7175.486024s 2024-03-25T18:11:34.184 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.183+0000 7ff8abfff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7ff8ac13f710 0x7ff8ac131c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:35294/0 (socket says 172.21.15.156:35294) 2024-03-25T18:11:34.184 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.183+0000 7ff8abfff640 1 -- 172.21.15.156:0/1883116033 learned_addr learned my addr 172.21.15.156:0/1883116033 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:11:34.189 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.188+0000 7ff8a97fa640 10 client.?.objecter ms_handle_connect 0x7ff8ac13f710 2024-03-25T18:11:34.189 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.188+0000 7ff8a97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:34.195 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.195+0000 7ff8a97fa640 10 client.?.objecter ms_handle_connect 0x7ff8ac1430c0 2024-03-25T18:11:34.195 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.195+0000 7ff8a97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:34.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.200+0000 7ff8a97fa640 10 client.?.objecter ms_handle_connect 0x7ff8ac0a98f0 2024-03-25T18:11:34.201 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.200+0000 7ff8a97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:34.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.203+0000 7ff8a97fa640 1 -- 172.21.15.156:0/1883116033 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1840522101 0 0) 0x7ff8ac138c80 con 0x7ff8ac13f710 2024-03-25T18:11:34.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.203+0000 7ff8a97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:11:34.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.203+0000 7ff8a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:34.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.203+0000 7ff8a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:34.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.203+0000 7ff8a97fa640 10 monclient(hunting): my global_id is 4369 2024-03-25T18:11:34.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.203+0000 7ff8a97fa640 1 -- 172.21.15.156:0/1883116033 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff890004eb0 con 0x7ff8ac13f710 2024-03-25T18:11:34.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.217+0000 7ff8a97fa640 1 -- 172.21.15.156:0/1883116033 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 842522033 0 0) 0x7ff8ac132a30 con 0x7ff8ac1430c0 2024-03-25T18:11:34.218 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.217+0000 7ff8a97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:11:34.218 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.217+0000 7ff8a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:34.218 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.217+0000 7ff8a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:34.218 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.217+0000 7ff8a97fa640 10 monclient(hunting): my global_id is 4371 2024-03-25T18:11:34.218 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.217+0000 7ff8a97fa640 1 -- 172.21.15.156:0/1883116033 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8ac138c80 con 0x7ff8ac1430c0 2024-03-25T18:11:34.286 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.285+0000 7ff8a97fa640 1 -- 172.21.15.156:0/1883116033 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2901494603 0 0) 0x7ff8ac138c80 con 0x7ff8ac1430c0 2024-03-25T18:11:34.286 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.285+0000 7ff8a97fa640 1 -- 172.21.15.156:0/1883116033 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8ac132a30 con 0x7ff8ac1430c0 2024-03-25T18:11:34.286 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.285+0000 7ff8a97fa640 1 -- 172.21.15.156:0/1883116033 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7ff8a4001b10 con 0x7ff8ac1430c0 2024-03-25T18:11:34.286 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.285+0000 7ff8a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:34.286 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.285+0000 7ff8a97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:34.286 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.285+0000 7ff8a97fa640 10 monclient(hunting): dump: 2024-03-25T18:11:34.286 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:34.287 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:34.287 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:34.287 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:34.287 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:34.287 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:34.287 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:34.287 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:34.287 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:34.287 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:34.287 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.285+0000 7ff8a97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:34.311 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.309+0000 7ff8a97fa640 1 -- 172.21.15.156:0/1883116033 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2020864171 0 0) 0x7ff8ac132a30 con 0x7ff8ac1430c0 2024-03-25T18:11:34.311 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.309+0000 7ff8a97fa640 1 -- 172.21.15.156:0/1883116033 >> v1:172.21.15.156:6790/0 conn(0x7ff8ac0a98f0 legacy=0x7ff8ac137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:34.311 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.309+0000 7ff8a97fa640 1 -- 172.21.15.156:0/1883116033 >> v1:172.21.15.194:6789/0 conn(0x7ff8ac13f710 legacy=0x7ff8ac131c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:34.311 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.309+0000 7ff8a97fa640 10 monclient: _finish_hunting 0 2024-03-25T18:11:34.311 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.309+0000 7ff8a97fa640 1 monclient: found mon.a 2024-03-25T18:11:34.311 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.309+0000 7ff8a97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:34.311 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.309+0000 7ff8a97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:11:34.311 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.309+0000 7ff8a97fa640 1 -- 172.21.15.156:0/1883116033 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ac139240 con 0x7ff8ac1430c0 2024-03-25T18:11:34.311 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.309+0000 7ff8a97fa640 10 monclient: _finish_auth 0 2024-03-25T18:11:34.311 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.309+0000 7ff8a97fa640 10 monclient: _check_auth_tickets 2024-03-25T18:11:34.311 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.309+0000 7ff8a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:34.311 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.309+0000 7ff8b2c72640 5 monclient: authenticate success, global_id 4371 2024-03-25T18:11:34.312 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.309+0000 7ff8b2c72640 10 monclient: _renew_subs 2024-03-25T18:11:34.312 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.309+0000 7ff8b2c72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:11:34.312 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.309+0000 7ff8b2c72640 1 -- 172.21.15.156:0/1883116033 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8ac136e50 con 0x7ff8ac1430c0 2024-03-25T18:11:34.312 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.309+0000 7ff8b2c72640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:34.312 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.309+0000 7ff8b2c72640 10 monclient: _renew_subs 2024-03-25T18:11:34.312 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.309+0000 7ff8b2c72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:11:34.312 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.309+0000 7ff8b2c72640 1 -- 172.21.15.156:0/1883116033 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8ac1389f0 con 0x7ff8ac1430c0 2024-03-25T18:11:34.312 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.309+0000 7ff8b2c72640 1 librados: init done 2024-03-25T18:11:34.312 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.310+0000 7ff8b2c72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:34.313 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.310+0000 7ff8b2c72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:34.313 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.310+0000 7ff8b2c72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:11:34.313 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.310+0000 7ff8b2c72640 1 -- 172.21.15.156:0/1883116033 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8ac0627b0 con 0x7ff8ac1430c0 2024-03-25T18:11:34.620 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.619+0000 7ff8a97fa640 1 -- 172.21.15.156:0/1883116033 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8a4002000 con 0x7ff8ac1430c0 2024-03-25T18:11:34.620 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.619+0000 7ff8a97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:34.620 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.619+0000 7ff8a97fa640 1 -- 172.21.15.156:0/1883116033 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7ff8a4003b30 con 0x7ff8ac1430c0 2024-03-25T18:11:34.620 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.619+0000 7ff8a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:34.620 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.619+0000 7ff8a97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:34.620 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.619+0000 7ff8a97fa640 10 monclient: dump: 2024-03-25T18:11:34.620 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:34.620 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:34.621 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:34.621 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:34.621 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:34.621 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:34.621 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:34.621 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:34.621 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:34.621 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:34.621 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.619+0000 7ff8aaffd640 4 set_mon_vals no callback set 2024-03-25T18:11:34.622 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.620+0000 7ff8a97fa640 1 -- 172.21.15.156:0/1883116033 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7ff8a4003d50 con 0x7ff8ac1430c0 2024-03-25T18:11:34.622 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.620+0000 7ff8b0a0e640 10 monclient: get_auth_request con 0x7ff89006dd90 auth_method 0 2024-03-25T18:11:34.622 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.621+0000 7ff8a97fa640 10 client.4371.objecter ms_handle_connect 0x7ff89006dd90 2024-03-25T18:11:34.627 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.626+0000 7ff8a97fa640 1 -- 172.21.15.156:0/1883116033 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7ff8a40536c0 con 0x7ff8ac1430c0 2024-03-25T18:11:34.627 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.626+0000 7ff8a97fa640 10 client.4371.objecter ms_dispatch 0x7ff8ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:34.627 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.626+0000 7ff8a97fa640 3 client.4371.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:34.627 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.626+0000 7ff8a97fa640 3 client.4371.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:34.627 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.626+0000 7ff8a97fa640 20 client.4371.objecter dump_active .. 0 homeless 2024-03-25T18:11:34.628 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.626+0000 7ff8a97fa640 1 -- 172.21.15.156:0/1883116033 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff8a404f1d0 con 0x7ff8ac1430c0 2024-03-25T18:11:34.628 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.626+0000 7ff8a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:34.628 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.626+0000 7ff8a97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:34.888 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.887+0000 7ff8b2c72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:34.889 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.888+0000 7ff8b2c72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:34.889 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.888+0000 7ff8b2c72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:11:34.889 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.888+0000 7ff8b2c72640 1 -- 172.21.15.156:0/1883116033 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8ac13dd60 con 0x7ff8ac1430c0 2024-03-25T18:11:34.937 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.936+0000 7ff8a97fa640 1 -- 172.21.15.156:0/1883116033 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7ff8a4051100 con 0x7ff8ac1430c0 2024-03-25T18:11:34.937 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.936+0000 7ff8a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:34.937 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.936+0000 7ff8a97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:34.938 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:11:34.938 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:34.940 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.939+0000 7ff8b2c72640 10 librados: watch_flush enter 2024-03-25T18:11:34.941 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.939+0000 7ff8b2c72640 10 librados: watch_flush exit 2024-03-25T18:11:34.941 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.939+0000 7ff8b2c72640 20 client.4371.objecter shutdown clearing up homeless session... 2024-03-25T18:11:34.941 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.939+0000 7ff8b2c72640 10 client.4371.objecter successfully canceled tick 2024-03-25T18:11:34.941 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.939+0000 7ff8b2c72640 1 -- 172.21.15.156:0/1883116033 >> v1:172.21.15.194:6800/2946427713 conn(0x7ff89006dd90 legacy=0x7ff890070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:34.941 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.939+0000 7ff8b2c72640 10 monclient: shutdown 2024-03-25T18:11:34.941 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.939+0000 7ff8b2c72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:34.941 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.939+0000 7ff8b2c72640 1 -- 172.21.15.156:0/1883116033 >> v1:172.21.15.156:6789/0 conn(0x7ff8ac1430c0 legacy=0x7ff8ac132320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:34.941 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.939+0000 7ff8b2c72640 1 -- 172.21.15.156:0/1883116033 shutdown_connections 2024-03-25T18:11:34.941 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.939+0000 7ff8b2c72640 1 -- 172.21.15.156:0/1883116033 >> 172.21.15.156:0/1883116033 conn(0x7ff8ac0b7ea0 msgr2=0x7ff8ac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:34.941 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.939+0000 7ff8b2c72640 1 -- 172.21.15.156:0/1883116033 shutdown_connections 2024-03-25T18:11:34.941 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.939+0000 7ff8b2c72640 1 -- 172.21.15.156:0/1883116033 wait complete. 2024-03-25T18:11:34.941 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:34.939+0000 7ff8b2c72640 1 librados: shutdown 2024-03-25T18:11:34.952 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:41.759 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:41.874 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.873+0000 7f5a4609d640 10 monclient: get_monmap_and_config 2024-03-25T18:11:41.874 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.873+0000 7f5a4609d640 10 monclient: build_initial_monmap 2024-03-25T18:11:41.874 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.873+0000 7f5a4609d640 10 monclient: monmap: 2024-03-25T18:11:41.874 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:11:41.874 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:41.874 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:11:41.874583+0000 2024-03-25T18:11:41.874 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:11:41.874583+0000 2024-03-25T18:11:41.874 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:41.874 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:41.875 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:11:41.875 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:11:41.875 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:11:41.875 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:41.876 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.874+0000 7f5a4609d640 1 Processor -- start 2024-03-25T18:11:41.876 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.874+0000 7f5a4609d640 1 -- start start 2024-03-25T18:11:41.876 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.874+0000 7f5a4609d640 10 monclient: init 2024-03-25T18:11:41.876 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.875+0000 7f5a4609d640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:41.876 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.875+0000 7f5a4609d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:11:41.876 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.875+0000 7f5a4609d640 10 monclient(hunting): picked mon.noname-a con 0x7f5a400705b0 addr v1:172.21.15.156:6789/0 2024-03-25T18:11:41.876 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.875+0000 7f5a4609d640 10 monclient(hunting): picked mon.noname-c con 0x7f5a401402d0 addr v1:172.21.15.194:6789/0 2024-03-25T18:11:41.876 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.875+0000 7f5a4609d640 10 monclient(hunting): picked mon.noname-b con 0x7f5a40149ae0 addr v1:172.21.15.156:6790/0 2024-03-25T18:11:41.876 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.875+0000 7f5a4609d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a400718c0 con 0x7f5a400705b0 2024-03-25T18:11:41.876 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.875+0000 7f5a4609d640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a400714c0 con 0x7f5a401402d0 2024-03-25T18:11:41.877 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.875+0000 7f5a4609d640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a40072b50 con 0x7f5a40149ae0 2024-03-25T18:11:41.877 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.875+0000 7f5a4609d640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:41.877 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.875+0000 7f5a4609d640 10 monclient(hunting): authenticate will time out at 7183.186232s 2024-03-25T18:11:41.881 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.881+0000 7f5a3f7fe640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f5a400705b0 0x7f5a4013d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:38808/0 (socket says 172.21.15.156:38808) 2024-03-25T18:11:41.881 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.881+0000 7f5a3f7fe640 1 -- 172.21.15.156:0/811512926 learned_addr learned my addr 172.21.15.156:0/811512926 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:11:41.896 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.895+0000 7f5a3e7fc640 1 -- 172.21.15.156:0/811512926 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823203939 0 0) 0x7f5a400714c0 con 0x7f5a401402d0 2024-03-25T18:11:41.896 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.895+0000 7f5a3e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:41.896 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.895+0000 7f5a3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:41.896 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.895+0000 7f5a3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:41.896 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.895+0000 7f5a3e7fc640 10 monclient(hunting): my global_id is 4372 2024-03-25T18:11:41.896 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.895+0000 7f5a3e7fc640 1 -- 172.21.15.156:0/811512926 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a28003520 con 0x7f5a401402d0 2024-03-25T18:11:41.898 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.897+0000 7f5a3e7fc640 1 -- 172.21.15.156:0/811512926 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1836492210 0 0) 0x7f5a400718c0 con 0x7f5a400705b0 2024-03-25T18:11:41.898 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.897+0000 7f5a3e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:41.898 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.897+0000 7f5a3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:41.898 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.897+0000 7f5a3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:41.898 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.897+0000 7f5a3e7fc640 10 monclient(hunting): my global_id is 4374 2024-03-25T18:11:41.898 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.897+0000 7f5a3e7fc640 1 -- 172.21.15.156:0/811512926 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a400714c0 con 0x7f5a400705b0 2024-03-25T18:11:41.907 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.906+0000 7f5a3e7fc640 1 -- 172.21.15.156:0/811512926 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4037243894 0 0) 0x7f5a400714c0 con 0x7f5a400705b0 2024-03-25T18:11:41.907 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.906+0000 7f5a3e7fc640 1 -- 172.21.15.156:0/811512926 >> v1:172.21.15.156:6790/0 conn(0x7f5a40149ae0 legacy=0x7f5a4014bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:41.907 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.906+0000 7f5a3e7fc640 1 -- 172.21.15.156:0/811512926 >> v1:172.21.15.194:6789/0 conn(0x7f5a401402d0 legacy=0x7f5a40140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:41.907 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.906+0000 7f5a3e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:11:41.908 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.906+0000 7f5a3e7fc640 1 monclient: found mon.noname-a 2024-03-25T18:11:41.908 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.906+0000 7f5a3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:41.908 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.906+0000 7f5a3e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:11:41.908 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.906+0000 7f5a3e7fc640 1 -- 172.21.15.156:0/811512926 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a4013dd10 con 0x7f5a400705b0 2024-03-25T18:11:41.908 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.906+0000 7f5a3e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:11:41.908 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.906+0000 7f5a3e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:11:41.908 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.906+0000 7f5a3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:41.908 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.906+0000 7f5a3e7fc640 1 -- 172.21.15.156:0/811512926 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5a30001670 con 0x7f5a400705b0 2024-03-25T18:11:41.908 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.906+0000 7f5a3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:41.908 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.906+0000 7f5a3e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:41.908 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.906+0000 7f5a3e7fc640 10 monclient: dump: 2024-03-25T18:11:41.908 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:41.908 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:41.908 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:41.908 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:41.908 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:41.909 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:41.909 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:41.909 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:41.909 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:41.909 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:41.909 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.906+0000 7f5a4609d640 5 monclient: authenticate success, global_id 4374 2024-03-25T18:11:41.909 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.906+0000 7f5a4609d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:41.928 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.927+0000 7f5a3e7fc640 1 -- 172.21.15.156:0/811512926 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a300017d0 con 0x7f5a400705b0 2024-03-25T18:11:41.928 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.927+0000 7f5a3e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:41.928 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.927+0000 7f5a3e7fc640 1 -- 172.21.15.156:0/811512926 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5a30001ba0 con 0x7f5a400705b0 2024-03-25T18:11:41.928 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.927+0000 7f5a4609d640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:41.928 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.927+0000 7f5a4609d640 4 set_mon_vals no callback set 2024-03-25T18:11:41.928 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.927+0000 7f5a3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:41.928 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.927+0000 7f5a3e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:41.928 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.927+0000 7f5a3e7fc640 10 monclient: dump: 2024-03-25T18:11:41.929 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:41.929 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:41.929 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:41.929 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:41.929 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:41.929 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:41.929 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:41.929 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:41.929 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:41.929 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:41.931 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.928+0000 7f5a4609d640 10 monclient: shutdown 2024-03-25T18:11:41.931 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.928+0000 7f5a4609d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:41.931 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.928+0000 7f5a4609d640 1 -- 172.21.15.156:0/811512926 >> v1:172.21.15.156:6789/0 conn(0x7f5a400705b0 legacy=0x7f5a4013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:41.932 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.928+0000 7f5a4609d640 1 -- 172.21.15.156:0/811512926 shutdown_connections 2024-03-25T18:11:41.932 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.928+0000 7f5a4609d640 1 -- 172.21.15.156:0/811512926 >> 172.21.15.156:0/811512926 conn(0x7f5a400b5d10 msgr2=0x7f5a400b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:41.932 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.928+0000 7f5a4609d640 1 -- 172.21.15.156:0/811512926 shutdown_connections 2024-03-25T18:11:41.932 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.928+0000 7f5a4609d640 1 -- 172.21.15.156:0/811512926 wait complete. 2024-03-25T18:11:41.932 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.928+0000 7f5a4609d640 10 monclient: build_initial_monmap 2024-03-25T18:11:41.932 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.928+0000 7f5a4609d640 10 monclient: monmap: 2024-03-25T18:11:41.932 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:11:41.932 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:41.932 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:11:41.932 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:11:41.932 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:41.932 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:11:41.932 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:11:41.932 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:11:41.932 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:11:41.932 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:41.933 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.929+0000 7f5a4609d640 1 librados: starting msgr at 2024-03-25T18:11:41.933 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.929+0000 7f5a4609d640 1 librados: starting objecter 2024-03-25T18:11:41.933 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.929+0000 7f5a4609d640 1 Processor -- start 2024-03-25T18:11:41.933 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.929+0000 7f5a4609d640 1 -- start start 2024-03-25T18:11:41.933 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.929+0000 7f5a4609d640 1 librados: setting wanted keys 2024-03-25T18:11:41.933 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.929+0000 7f5a4609d640 1 librados: calling monclient init 2024-03-25T18:11:41.933 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.929+0000 7f5a4609d640 10 monclient: init 2024-03-25T18:11:41.933 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.930+0000 7f5a4609d640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:41.933 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.930+0000 7f5a4609d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:11:41.933 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.930+0000 7f5a4609d640 10 monclient(hunting): picked mon.noname-a con 0x7f5a400705b0 addr v1:172.21.15.156:6789/0 2024-03-25T18:11:41.933 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.930+0000 7f5a4609d640 10 monclient(hunting): picked mon.noname-b con 0x7f5a401402d0 addr v1:172.21.15.194:6789/0 2024-03-25T18:11:41.933 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.930+0000 7f5a4609d640 10 monclient(hunting): picked mon.noname-c con 0x7f5a40149ae0 addr v1:172.21.15.156:6790/0 2024-03-25T18:11:41.933 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.930+0000 7f5a4609d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4013e780 con 0x7f5a400705b0 2024-03-25T18:11:41.934 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.930+0000 7f5a4609d640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a40134870 con 0x7f5a401402d0 2024-03-25T18:11:41.934 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.930+0000 7f5a4609d640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a401360f0 con 0x7f5a40149ae0 2024-03-25T18:11:41.934 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.930+0000 7f5a4609d640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:41.934 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.930+0000 7f5a4609d640 10 monclient(hunting): authenticate will time out at 7183.241051s 2024-03-25T18:11:41.939 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.938+0000 7f5a3effd640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f5a401402d0 0x7f5a40133a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:40134/0 (socket says 172.21.15.156:40134) 2024-03-25T18:11:41.939 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.938+0000 7f5a3effd640 1 -- 172.21.15.156:0/1297092187 learned_addr learned my addr 172.21.15.156:0/1297092187 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:11:41.940 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.940+0000 7f5a27fff640 10 client.?.objecter ms_handle_connect 0x7f5a401402d0 2024-03-25T18:11:41.940 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.940+0000 7f5a27fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:41.942 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.942+0000 7f5a27fff640 10 client.?.objecter ms_handle_connect 0x7f5a400705b0 2024-03-25T18:11:41.942 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.942+0000 7f5a27fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:41.946 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.945+0000 7f5a27fff640 10 client.?.objecter ms_handle_connect 0x7f5a40149ae0 2024-03-25T18:11:41.946 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.945+0000 7f5a27fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:41.946 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.946+0000 7f5a27fff640 1 -- 172.21.15.156:0/1297092187 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765706440 0 0) 0x7f5a40134870 con 0x7f5a401402d0 2024-03-25T18:11:41.946 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.946+0000 7f5a27fff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:41.947 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.946+0000 7f5a27fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:41.947 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.946+0000 7f5a27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:41.947 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.946+0000 7f5a27fff640 10 monclient(hunting): my global_id is 4375 2024-03-25T18:11:41.947 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.946+0000 7f5a27fff640 1 -- 172.21.15.156:0/1297092187 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a2c004eb0 con 0x7f5a401402d0 2024-03-25T18:11:41.954 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.953+0000 7f5a27fff640 1 -- 172.21.15.156:0/1297092187 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3830597007 0 0) 0x7f5a2c004eb0 con 0x7f5a401402d0 2024-03-25T18:11:41.954 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.953+0000 7f5a27fff640 1 -- 172.21.15.156:0/1297092187 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a40134870 con 0x7f5a401402d0 2024-03-25T18:11:41.954 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.953+0000 7f5a27fff640 1 -- 172.21.15.156:0/1297092187 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5a34001a10 con 0x7f5a401402d0 2024-03-25T18:11:41.954 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.953+0000 7f5a27fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:41.954 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.953+0000 7f5a27fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:41.954 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.953+0000 7f5a27fff640 10 monclient(hunting): dump: 2024-03-25T18:11:41.954 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:41.954 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:41.954 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:41.955 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:41.955 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:41.955 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:41.955 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:41.955 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:41.955 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:41.955 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:41.955 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.953+0000 7f5a27fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:41.955 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.954+0000 7f5a27fff640 1 -- 172.21.15.156:0/1297092187 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4165608435 0 0) 0x7f5a4013e780 con 0x7f5a400705b0 2024-03-25T18:11:41.956 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.954+0000 7f5a27fff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:41.956 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.954+0000 7f5a27fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:41.956 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.954+0000 7f5a27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:41.956 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.954+0000 7f5a27fff640 10 monclient(hunting): my global_id is 4377 2024-03-25T18:11:41.956 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.954+0000 7f5a27fff640 1 -- 172.21.15.156:0/1297092187 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a2c004eb0 con 0x7f5a400705b0 2024-03-25T18:11:41.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.961+0000 7f5a27fff640 1 -- 172.21.15.156:0/1297092187 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3760793735 0 0) 0x7f5a401360f0 con 0x7f5a40149ae0 2024-03-25T18:11:41.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.961+0000 7f5a27fff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:41.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.961+0000 7f5a27fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:41.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.961+0000 7f5a27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:41.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.961+0000 7f5a27fff640 10 monclient(hunting): my global_id is 4385 2024-03-25T18:11:41.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.961+0000 7f5a27fff640 1 -- 172.21.15.156:0/1297092187 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a4013e780 con 0x7f5a40149ae0 2024-03-25T18:11:41.976 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.975+0000 7f5a27fff640 1 -- 172.21.15.156:0/1297092187 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2016929722 0 0) 0x7f5a2c004eb0 con 0x7f5a400705b0 2024-03-25T18:11:41.976 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.975+0000 7f5a27fff640 1 -- 172.21.15.156:0/1297092187 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a401360f0 con 0x7f5a400705b0 2024-03-25T18:11:41.976 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.975+0000 7f5a27fff640 1 -- 172.21.15.156:0/1297092187 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5a30001ae0 con 0x7f5a400705b0 2024-03-25T18:11:41.976 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.975+0000 7f5a27fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:41.976 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.975+0000 7f5a27fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:41.976 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.975+0000 7f5a27fff640 10 monclient(hunting): dump: 2024-03-25T18:11:41.976 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:41.976 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:41.976 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:41.976 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:41.976 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:41.976 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:41.976 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:41.976 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:41.976 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:41.977 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:41.979 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.978+0000 7f5a27fff640 1 -- 172.21.15.156:0/1297092187 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 460627377 0 0) 0x7f5a40134870 con 0x7f5a401402d0 2024-03-25T18:11:41.979 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.978+0000 7f5a27fff640 1 -- 172.21.15.156:0/1297092187 >> v1:172.21.15.156:6790/0 conn(0x7f5a40149ae0 legacy=0x7f5a40134160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:41.979 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.978+0000 7f5a27fff640 1 -- 172.21.15.156:0/1297092187 >> v1:172.21.15.156:6789/0 conn(0x7f5a400705b0 legacy=0x7f5a401d7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:41.979 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.978+0000 7f5a27fff640 10 monclient: _finish_hunting 0 2024-03-25T18:11:41.980 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.978+0000 7f5a27fff640 1 monclient: found mon.b 2024-03-25T18:11:41.980 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.978+0000 7f5a27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:41.980 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.978+0000 7f5a27fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:11:41.980 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.978+0000 7f5a27fff640 1 -- 172.21.15.156:0/1297092187 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a40136280 con 0x7f5a401402d0 2024-03-25T18:11:41.980 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.978+0000 7f5a27fff640 10 monclient: _finish_auth 0 2024-03-25T18:11:41.980 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.978+0000 7f5a27fff640 10 monclient: _check_auth_tickets 2024-03-25T18:11:41.980 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.978+0000 7f5a27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:41.980 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.978+0000 7f5a4609d640 5 monclient: authenticate success, global_id 4375 2024-03-25T18:11:41.980 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.978+0000 7f5a4609d640 10 monclient: _renew_subs 2024-03-25T18:11:41.980 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.978+0000 7f5a4609d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:11:41.980 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.978+0000 7f5a4609d640 1 -- 172.21.15.156:0/1297092187 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a401391f0 con 0x7f5a401402d0 2024-03-25T18:11:41.980 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.978+0000 7f5a4609d640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:41.981 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.978+0000 7f5a4609d640 10 monclient: _renew_subs 2024-03-25T18:11:41.981 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.978+0000 7f5a4609d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:11:41.981 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.978+0000 7f5a4609d640 1 -- 172.21.15.156:0/1297092187 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a4013dfd0 con 0x7f5a401402d0 2024-03-25T18:11:41.981 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.978+0000 7f5a4609d640 1 librados: init done 2024-03-25T18:11:41.981 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.979+0000 7f5a4609d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:41.981 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.979+0000 7f5a4609d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:41.981 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.979+0000 7f5a4609d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:11:41.981 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:41.979+0000 7f5a4609d640 1 -- 172.21.15.156:0/1297092187 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a40136470 con 0x7f5a401402d0 2024-03-25T18:11:42.259 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.258+0000 7f5a27fff640 1 -- 172.21.15.156:0/1297092187 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a34001c90 con 0x7f5a401402d0 2024-03-25T18:11:42.260 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.258+0000 7f5a27fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:42.260 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.258+0000 7f5a27fff640 1 -- 172.21.15.156:0/1297092187 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5a34003a60 con 0x7f5a401402d0 2024-03-25T18:11:42.260 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.258+0000 7f5a27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:42.260 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.258+0000 7f5a27fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:42.260 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.258+0000 7f5a27fff640 10 monclient: dump: 2024-03-25T18:11:42.260 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:42.260 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:42.260 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:42.260 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:42.260 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:42.261 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:42.261 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:42.261 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:42.261 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:42.261 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:42.261 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.258+0000 7f5a3dffb640 4 set_mon_vals no callback set 2024-03-25T18:11:42.261 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.259+0000 7f5a27fff640 1 -- 172.21.15.156:0/1297092187 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f5a34003c80 con 0x7f5a401402d0 2024-03-25T18:11:42.262 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.259+0000 7f5a3f7fe640 10 monclient: get_auth_request con 0x7f5a2c06e020 auth_method 0 2024-03-25T18:11:42.262 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.259+0000 7f5a27fff640 10 client.4375.objecter ms_handle_connect 0x7f5a2c06e020 2024-03-25T18:11:42.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.266+0000 7f5a27fff640 1 -- 172.21.15.156:0/1297092187 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f5a34054870 con 0x7f5a401402d0 2024-03-25T18:11:42.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.266+0000 7f5a27fff640 10 client.4375.objecter ms_dispatch 0x7f5a40000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:42.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.266+0000 7f5a27fff640 3 client.4375.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:42.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.266+0000 7f5a27fff640 3 client.4375.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:42.268 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.266+0000 7f5a27fff640 20 client.4375.objecter dump_active .. 0 homeless 2024-03-25T18:11:42.268 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.266+0000 7f5a27fff640 1 -- 172.21.15.156:0/1297092187 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a34050380 con 0x7f5a401402d0 2024-03-25T18:11:42.268 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.266+0000 7f5a27fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:42.268 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.266+0000 7f5a27fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:42.491 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.490+0000 7f5a4609d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:42.492 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.490+0000 7f5a4609d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:42.492 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.490+0000 7f5a4609d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:11:42.492 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.490+0000 7f5a4609d640 1 -- 172.21.15.156:0/1297092187 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a400627b0 con 0x7f5a401402d0 2024-03-25T18:11:42.537 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.536+0000 7f5a27fff640 1 -- 172.21.15.156:0/1297092187 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f5a340522b0 con 0x7f5a401402d0 2024-03-25T18:11:42.537 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.536+0000 7f5a27fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:42.537 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.536+0000 7f5a27fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:42.538 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:11:42.538 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:42.540 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.538+0000 7f5a4609d640 10 librados: watch_flush enter 2024-03-25T18:11:42.540 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.538+0000 7f5a4609d640 10 librados: watch_flush exit 2024-03-25T18:11:42.541 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.538+0000 7f5a4609d640 20 client.4375.objecter shutdown clearing up homeless session... 2024-03-25T18:11:42.541 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.538+0000 7f5a4609d640 10 client.4375.objecter successfully canceled tick 2024-03-25T18:11:42.541 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.539+0000 7f5a4609d640 1 -- 172.21.15.156:0/1297092187 >> v1:172.21.15.194:6800/2946427713 conn(0x7f5a2c06e020 legacy=0x7f5a2c0704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:42.541 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.539+0000 7f5a4609d640 10 monclient: shutdown 2024-03-25T18:11:42.541 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.539+0000 7f5a4609d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:42.541 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.539+0000 7f5a4609d640 1 -- 172.21.15.156:0/1297092187 >> v1:172.21.15.194:6789/0 conn(0x7f5a401402d0 legacy=0x7f5a40133a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:42.541 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.539+0000 7f5a4609d640 1 -- 172.21.15.156:0/1297092187 shutdown_connections 2024-03-25T18:11:42.541 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.539+0000 7f5a4609d640 1 -- 172.21.15.156:0/1297092187 >> 172.21.15.156:0/1297092187 conn(0x7f5a400b5d10 msgr2=0x7f5a400b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:42.541 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.539+0000 7f5a4609d640 1 -- 172.21.15.156:0/1297092187 shutdown_connections 2024-03-25T18:11:42.541 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.539+0000 7f5a4609d640 1 -- 172.21.15.156:0/1297092187 wait complete. 2024-03-25T18:11:42.541 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:42.539+0000 7f5a4609d640 1 librados: shutdown 2024-03-25T18:11:42.552 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:43.691 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:03:49.682 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:43.742 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:03:49.732 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:43.778 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:03:49.769 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:43.796 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:03:49.763 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:43.802 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:03:49.769 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:43.808 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:03:49.798 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:43.818 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:03:49.809 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:43.830 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:03:49.797 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:43.857 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:03:49.847 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:49.358 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:49.475 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.474+0000 7f2858d0e640 10 monclient: get_monmap_and_config 2024-03-25T18:11:49.476 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.474+0000 7f2858d0e640 10 monclient: build_initial_monmap 2024-03-25T18:11:49.476 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.475+0000 7f2858d0e640 10 monclient: monmap: 2024-03-25T18:11:49.476 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:11:49.476 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:49.476 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:11:49.476010+0000 2024-03-25T18:11:49.476 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:11:49.476010+0000 2024-03-25T18:11:49.476 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:49.476 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:49.476 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:11:49.476 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:11:49.476 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:11:49.476 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:49.477 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.475+0000 7f2858d0e640 1 Processor -- start 2024-03-25T18:11:49.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.475+0000 7f2858d0e640 1 -- start start 2024-03-25T18:11:49.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.475+0000 7f2858d0e640 10 monclient: init 2024-03-25T18:11:49.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.476+0000 7f2858d0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:49.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.477+0000 7f2858d0e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:11:49.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.477+0000 7f2858d0e640 10 monclient(hunting): picked mon.noname-a con 0x7f28540a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:11:49.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.477+0000 7f2858d0e640 10 monclient(hunting): picked mon.noname-c con 0x7f285413f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:11:49.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.477+0000 7f2858d0e640 10 monclient(hunting): picked mon.noname-b con 0x7f28541430c0 addr v1:172.21.15.156:6790/0 2024-03-25T18:11:49.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.477+0000 7f2858d0e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2854142520 con 0x7f28540a98f0 2024-03-25T18:11:49.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.477+0000 7f2858d0e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28540b28a0 con 0x7f285413f710 2024-03-25T18:11:49.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.477+0000 7f2858d0e640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28540b2a30 con 0x7f28541430c0 2024-03-25T18:11:49.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.477+0000 7f2858d0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:49.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.477+0000 7f2858d0e640 10 monclient(hunting): authenticate will time out at 7190.787951s 2024-03-25T18:11:49.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.482+0000 7f285259c640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f28540a98f0 0x7f28540a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:43350/0 (socket says 172.21.15.156:43350) 2024-03-25T18:11:49.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.482+0000 7f285259c640 1 -- 172.21.15.156:0/1695725857 learned_addr learned my addr 172.21.15.156:0/1695725857 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:11:49.496 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.495+0000 7f285159a640 1 -- 172.21.15.156:0/1695725857 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615768740 0 0) 0x7f2854142520 con 0x7f28540a98f0 2024-03-25T18:11:49.496 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.495+0000 7f285159a640 10 monclient(hunting): _negotiate 2024-03-25T18:11:49.496 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.495+0000 7f285159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:49.496 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.495+0000 7f285159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:49.496 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.495+0000 7f285159a640 10 monclient(hunting): my global_id is 4380 2024-03-25T18:11:49.496 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.495+0000 7f285159a640 1 -- 172.21.15.156:0/1695725857 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f283c003520 con 0x7f28540a98f0 2024-03-25T18:11:49.502 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.501+0000 7f285159a640 1 -- 172.21.15.156:0/1695725857 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2106651268 0 0) 0x7f28540b28a0 con 0x7f285413f710 2024-03-25T18:11:49.502 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.501+0000 7f285159a640 10 monclient(hunting): _negotiate 2024-03-25T18:11:49.502 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.501+0000 7f285159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:49.502 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.501+0000 7f285159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:49.503 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.501+0000 7f285159a640 10 monclient(hunting): my global_id is 4378 2024-03-25T18:11:49.503 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.501+0000 7f285159a640 1 -- 172.21.15.156:0/1695725857 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2854142520 con 0x7f285413f710 2024-03-25T18:11:49.505 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.504+0000 7f285159a640 1 -- 172.21.15.156:0/1695725857 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 222679430 0 0) 0x7f28540b2a30 con 0x7f28541430c0 2024-03-25T18:11:49.505 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.504+0000 7f285159a640 10 monclient(hunting): _negotiate 2024-03-25T18:11:49.505 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.504+0000 7f285159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:49.505 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.504+0000 7f285159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:49.505 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.504+0000 7f285159a640 10 monclient(hunting): my global_id is 4388 2024-03-25T18:11:49.505 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.504+0000 7f285159a640 1 -- 172.21.15.156:0/1695725857 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28540b28a0 con 0x7f28541430c0 2024-03-25T18:11:49.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.507+0000 7f285159a640 1 -- 172.21.15.156:0/1695725857 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4259591773 0 0) 0x7f283c003520 con 0x7f28540a98f0 2024-03-25T18:11:49.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.507+0000 7f285159a640 1 -- 172.21.15.156:0/1695725857 >> v1:172.21.15.156:6790/0 conn(0x7f28541430c0 legacy=0x7f285414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:49.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.507+0000 7f285159a640 1 -- 172.21.15.156:0/1695725857 >> v1:172.21.15.194:6789/0 conn(0x7f285413f710 legacy=0x7f28541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:49.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.507+0000 7f285159a640 10 monclient: _finish_hunting 0 2024-03-25T18:11:49.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.507+0000 7f285159a640 1 monclient: found mon.noname-a 2024-03-25T18:11:49.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.507+0000 7f285159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:49.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.507+0000 7f285159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:11:49.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.507+0000 7f285159a640 1 -- 172.21.15.156:0/1695725857 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28540b44c0 con 0x7f28540a98f0 2024-03-25T18:11:49.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.507+0000 7f285159a640 10 monclient: _finish_auth 0 2024-03-25T18:11:49.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.507+0000 7f285159a640 10 monclient: _check_auth_tickets 2024-03-25T18:11:49.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.507+0000 7f285159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:49.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.507+0000 7f285159a640 1 -- 172.21.15.156:0/1695725857 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f2844001690 con 0x7f28540a98f0 2024-03-25T18:11:49.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.507+0000 7f285159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:49.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.507+0000 7f285159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:49.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.507+0000 7f285159a640 10 monclient: dump: 2024-03-25T18:11:49.509 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:49.509 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:49.509 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:49.509 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:49.509 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:49.509 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:49.509 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:49.509 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:49.509 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:49.509 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:49.510 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.507+0000 7f2858d0e640 5 monclient: authenticate success, global_id 4380 2024-03-25T18:11:49.510 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.507+0000 7f2858d0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:49.518 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.518+0000 7f285159a640 1 -- 172.21.15.156:0/1695725857 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28440017f0 con 0x7f28540a98f0 2024-03-25T18:11:49.519 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.518+0000 7f285159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:49.519 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.518+0000 7f285159a640 1 -- 172.21.15.156:0/1695725857 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f2844001c00 con 0x7f28540a98f0 2024-03-25T18:11:49.519 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.518+0000 7f285159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:49.519 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.518+0000 7f285159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:49.519 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.518+0000 7f285159a640 10 monclient: dump: 2024-03-25T18:11:49.519 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:49.519 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:49.519 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:49.519 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:49.519 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:49.519 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:49.519 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:49.519 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:49.519 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:49.520 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:49.520 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.518+0000 7f2858d0e640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:49.520 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.518+0000 7f2858d0e640 4 set_mon_vals no callback set 2024-03-25T18:11:49.521 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.518+0000 7f2858d0e640 10 monclient: shutdown 2024-03-25T18:11:49.521 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.518+0000 7f2858d0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:49.521 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.518+0000 7f2858d0e640 1 -- 172.21.15.156:0/1695725857 >> v1:172.21.15.156:6789/0 conn(0x7f28540a98f0 legacy=0x7f28540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:49.521 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.519+0000 7f2858d0e640 1 -- 172.21.15.156:0/1695725857 shutdown_connections 2024-03-25T18:11:49.521 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.519+0000 7f2858d0e640 1 -- 172.21.15.156:0/1695725857 >> 172.21.15.156:0/1695725857 conn(0x7f28540b7ea0 msgr2=0x7f28540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:49.521 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.519+0000 7f2858d0e640 1 -- 172.21.15.156:0/1695725857 shutdown_connections 2024-03-25T18:11:49.521 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.519+0000 7f2858d0e640 1 -- 172.21.15.156:0/1695725857 wait complete. 2024-03-25T18:11:49.522 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.519+0000 7f2858d0e640 10 monclient: build_initial_monmap 2024-03-25T18:11:49.522 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.519+0000 7f2858d0e640 10 monclient: monmap: 2024-03-25T18:11:49.522 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:11:49.522 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:49.522 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:11:49.522 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:11:49.522 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:49.522 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:11:49.522 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:11:49.522 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:11:49.522 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:11:49.522 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:49.522 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.520+0000 7f2858d0e640 1 librados: starting msgr at 2024-03-25T18:11:49.523 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.520+0000 7f2858d0e640 1 librados: starting objecter 2024-03-25T18:11:49.523 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.520+0000 7f2858d0e640 1 Processor -- start 2024-03-25T18:11:49.523 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.520+0000 7f2858d0e640 1 -- start start 2024-03-25T18:11:49.523 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.520+0000 7f2858d0e640 1 librados: setting wanted keys 2024-03-25T18:11:49.523 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.520+0000 7f2858d0e640 1 librados: calling monclient init 2024-03-25T18:11:49.523 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.520+0000 7f2858d0e640 10 monclient: init 2024-03-25T18:11:49.524 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.521+0000 7f2858d0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:49.524 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.521+0000 7f2858d0e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:11:49.524 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.521+0000 7f2858d0e640 10 monclient(hunting): picked mon.noname-a con 0x7f28540a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:11:49.524 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.521+0000 7f2858d0e640 10 monclient(hunting): picked mon.noname-b con 0x7f285413f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:11:49.524 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.521+0000 7f2858d0e640 10 monclient(hunting): picked mon.noname-c con 0x7f28541430c0 addr v1:172.21.15.156:6790/0 2024-03-25T18:11:49.524 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.521+0000 7f2858d0e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28541db2f0 con 0x7f28540a98f0 2024-03-25T18:11:49.525 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.521+0000 7f2858d0e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2854139180 con 0x7f285413f710 2024-03-25T18:11:49.525 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.521+0000 7f2858d0e640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2854137670 con 0x7f28541430c0 2024-03-25T18:11:49.525 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.521+0000 7f2858d0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:49.525 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.521+0000 7f2858d0e640 10 monclient(hunting): authenticate will time out at 7190.832710s 2024-03-25T18:11:49.527 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.526+0000 7f2851d9b640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f285413f710 0x7f2854132f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:44684/0 (socket says 172.21.15.156:44684) 2024-03-25T18:11:49.527 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.526+0000 7f2851d9b640 1 -- 172.21.15.156:0/2718967444 learned_addr learned my addr 172.21.15.156:0/2718967444 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:11:49.530 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.529+0000 7f2842ffd640 10 client.?.objecter ms_handle_connect 0x7f285413f710 2024-03-25T18:11:49.530 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.529+0000 7f2842ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:49.541 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.540+0000 7f2842ffd640 10 client.?.objecter ms_handle_connect 0x7f28541430c0 2024-03-25T18:11:49.541 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.540+0000 7f2842ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:49.541 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.541+0000 7f2842ffd640 10 client.?.objecter ms_handle_connect 0x7f28540a98f0 2024-03-25T18:11:49.542 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.541+0000 7f2842ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:49.550 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.549+0000 7f2842ffd640 1 -- 172.21.15.156:0/2718967444 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434924008 0 0) 0x7f28541db2f0 con 0x7f28540a98f0 2024-03-25T18:11:49.550 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.549+0000 7f2842ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:49.550 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.549+0000 7f2842ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:49.550 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.549+0000 7f2842ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:49.550 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.549+0000 7f2842ffd640 10 monclient(hunting): my global_id is 4383 2024-03-25T18:11:49.550 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.549+0000 7f2842ffd640 1 -- 172.21.15.156:0/2718967444 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2834004eb0 con 0x7f28540a98f0 2024-03-25T18:11:49.552 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.551+0000 7f2842ffd640 1 -- 172.21.15.156:0/2718967444 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1128900964 0 0) 0x7f2854139180 con 0x7f285413f710 2024-03-25T18:11:49.552 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.551+0000 7f2842ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:49.553 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.551+0000 7f2842ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:49.553 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.551+0000 7f2842ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:49.553 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.551+0000 7f2842ffd640 10 monclient(hunting): my global_id is 4381 2024-03-25T18:11:49.553 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.551+0000 7f2842ffd640 1 -- 172.21.15.156:0/2718967444 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28541db2f0 con 0x7f285413f710 2024-03-25T18:11:49.560 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.559+0000 7f2842ffd640 1 -- 172.21.15.156:0/2718967444 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3640331136 0 0) 0x7f28541db2f0 con 0x7f285413f710 2024-03-25T18:11:49.560 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.559+0000 7f2842ffd640 1 -- 172.21.15.156:0/2718967444 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2854139180 con 0x7f285413f710 2024-03-25T18:11:49.560 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.559+0000 7f2842ffd640 1 -- 172.21.15.156:0/2718967444 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f28480019b0 con 0x7f285413f710 2024-03-25T18:11:49.560 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.559+0000 7f2842ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:49.560 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.559+0000 7f2842ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:49.560 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.559+0000 7f2842ffd640 10 monclient(hunting): dump: 2024-03-25T18:11:49.560 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:49.560 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:49.560 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:49.561 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:49.561 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:49.561 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:49.561 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:49.561 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:49.561 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:49.561 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:49.561 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.559+0000 7f2842ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:49.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.563+0000 7f2842ffd640 1 -- 172.21.15.156:0/2718967444 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1023628087 0 0) 0x7f2854137670 con 0x7f28541430c0 2024-03-25T18:11:49.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.563+0000 7f2842ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:49.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.563+0000 7f2842ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:49.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.563+0000 7f2842ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:49.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.563+0000 7f2842ffd640 10 monclient(hunting): my global_id is 4391 2024-03-25T18:11:49.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.563+0000 7f2842ffd640 1 -- 172.21.15.156:0/2718967444 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28541db2f0 con 0x7f28541430c0 2024-03-25T18:11:49.568 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.567+0000 7f2842ffd640 1 -- 172.21.15.156:0/2718967444 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1147937176 0 0) 0x7f2834004eb0 con 0x7f28540a98f0 2024-03-25T18:11:49.568 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.567+0000 7f2842ffd640 1 -- 172.21.15.156:0/2718967444 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2854137670 con 0x7f28540a98f0 2024-03-25T18:11:49.568 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.567+0000 7f2842ffd640 1 -- 172.21.15.156:0/2718967444 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f2844001b00 con 0x7f28540a98f0 2024-03-25T18:11:49.568 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.567+0000 7f2842ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:49.569 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.567+0000 7f2842ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:49.569 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.567+0000 7f2842ffd640 10 monclient(hunting): dump: 2024-03-25T18:11:49.569 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:49.569 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:49.569 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:49.569 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:49.569 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:49.569 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:49.569 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:49.569 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:49.569 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:49.569 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:49.578 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.576+0000 7f2842ffd640 1 -- 172.21.15.156:0/2718967444 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4023143315 0 0) 0x7f2854139180 con 0x7f285413f710 2024-03-25T18:11:49.578 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.576+0000 7f2842ffd640 1 -- 172.21.15.156:0/2718967444 >> v1:172.21.15.156:6790/0 conn(0x7f28541430c0 legacy=0x7f28541e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:49.578 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.576+0000 7f2842ffd640 1 -- 172.21.15.156:0/2718967444 >> v1:172.21.15.156:6789/0 conn(0x7f28540a98f0 legacy=0x7f28541dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:49.578 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.576+0000 7f2842ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:11:49.578 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.576+0000 7f2842ffd640 1 monclient: found mon.b 2024-03-25T18:11:49.578 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.576+0000 7f2842ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:49.578 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.576+0000 7f2842ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:11:49.578 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.576+0000 7f2842ffd640 1 -- 172.21.15.156:0/2718967444 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2854137200 con 0x7f285413f710 2024-03-25T18:11:49.578 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.576+0000 7f2842ffd640 10 monclient: _finish_auth 0 2024-03-25T18:11:49.578 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.576+0000 7f2842ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:11:49.578 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.576+0000 7f2842ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:49.578 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.576+0000 7f2858d0e640 5 monclient: authenticate success, global_id 4381 2024-03-25T18:11:49.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.576+0000 7f2858d0e640 10 monclient: _renew_subs 2024-03-25T18:11:49.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.577+0000 7f2858d0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:11:49.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.577+0000 7f2858d0e640 1 -- 172.21.15.156:0/2718967444 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28540b7290 con 0x7f285413f710 2024-03-25T18:11:49.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.577+0000 7f2858d0e640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:49.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.577+0000 7f2858d0e640 10 monclient: _renew_subs 2024-03-25T18:11:49.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.577+0000 7f2858d0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:11:49.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.577+0000 7f2858d0e640 1 -- 172.21.15.156:0/2718967444 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2854133680 con 0x7f285413f710 2024-03-25T18:11:49.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.577+0000 7f2858d0e640 1 librados: init done 2024-03-25T18:11:49.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.578+0000 7f2858d0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:49.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.578+0000 7f2858d0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:49.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.578+0000 7f2858d0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:11:49.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.578+0000 7f2858d0e640 1 -- 172.21.15.156:0/2718967444 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28540627b0 con 0x7f285413f710 2024-03-25T18:11:49.852 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.851+0000 7f2842ffd640 1 -- 172.21.15.156:0/2718967444 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2848003610 con 0x7f285413f710 2024-03-25T18:11:49.852 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.851+0000 7f2842ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:49.852 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.851+0000 7f2842ffd640 1 -- 172.21.15.156:0/2718967444 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f28480039e0 con 0x7f285413f710 2024-03-25T18:11:49.852 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.851+0000 7f2842ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:49.852 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.851+0000 7f2842ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:49.852 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.851+0000 7f2842ffd640 10 monclient: dump: 2024-03-25T18:11:49.852 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:49.852 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:49.852 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:49.852 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:49.852 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:49.852 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:49.853 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:49.853 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:49.853 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:49.853 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:49.853 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.851+0000 7f2850d99640 4 set_mon_vals no callback set 2024-03-25T18:11:49.853 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.851+0000 7f2842ffd640 1 -- 172.21.15.156:0/2718967444 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f2848003bc0 con 0x7f285413f710 2024-03-25T18:11:49.853 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.852+0000 7f285259c640 10 monclient: get_auth_request con 0x7f283406df60 auth_method 0 2024-03-25T18:11:49.853 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.852+0000 7f2842ffd640 10 client.4381.objecter ms_handle_connect 0x7f283406df60 2024-03-25T18:11:49.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.857+0000 7f2842ffd640 1 -- 172.21.15.156:0/2718967444 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f2848054840 con 0x7f285413f710 2024-03-25T18:11:49.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.857+0000 7f2842ffd640 10 client.4381.objecter ms_dispatch 0x7f2854000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:49.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.857+0000 7f2842ffd640 3 client.4381.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:49.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.857+0000 7f2842ffd640 3 client.4381.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:49.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.857+0000 7f2842ffd640 20 client.4381.objecter dump_active .. 0 homeless 2024-03-25T18:11:49.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.858+0000 7f2842ffd640 1 -- 172.21.15.156:0/2718967444 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2848050350 con 0x7f285413f710 2024-03-25T18:11:49.859 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.858+0000 7f2842ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:49.859 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:49.858+0000 7f2842ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:50.113 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:50.112+0000 7f2858d0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:50.114 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:50.112+0000 7f2858d0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:50.114 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:50.112+0000 7f2858d0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:11:50.114 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:50.112+0000 7f2858d0e640 1 -- 172.21.15.156:0/2718967444 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f285413e2b0 con 0x7f285413f710 2024-03-25T18:11:50.154 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:50.153+0000 7f2842ffd640 1 -- 172.21.15.156:0/2718967444 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f2848052280 con 0x7f285413f710 2024-03-25T18:11:50.154 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:50.153+0000 7f2842ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:50.154 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:50.153+0000 7f2842ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:50.154 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:11:50.155 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:50.157 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:50.155+0000 7f2858d0e640 10 librados: watch_flush enter 2024-03-25T18:11:50.157 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:50.155+0000 7f2858d0e640 10 librados: watch_flush exit 2024-03-25T18:11:50.157 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:50.155+0000 7f2858d0e640 20 client.4381.objecter shutdown clearing up homeless session... 2024-03-25T18:11:50.157 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:50.155+0000 7f2858d0e640 10 client.4381.objecter successfully canceled tick 2024-03-25T18:11:50.157 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:50.156+0000 7f2858d0e640 1 -- 172.21.15.156:0/2718967444 >> v1:172.21.15.194:6800/2946427713 conn(0x7f283406df60 legacy=0x7f2834070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:50.157 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:50.156+0000 7f2858d0e640 10 monclient: shutdown 2024-03-25T18:11:50.157 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:50.156+0000 7f2858d0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:50.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:50.156+0000 7f2858d0e640 1 -- 172.21.15.156:0/2718967444 >> v1:172.21.15.194:6789/0 conn(0x7f285413f710 legacy=0x7f2854132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:50.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:50.156+0000 7f2858d0e640 1 -- 172.21.15.156:0/2718967444 shutdown_connections 2024-03-25T18:11:50.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:50.156+0000 7f2858d0e640 1 -- 172.21.15.156:0/2718967444 >> 172.21.15.156:0/2718967444 conn(0x7f28540b7ea0 msgr2=0x7f2854074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:50.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:50.156+0000 7f2858d0e640 1 -- 172.21.15.156:0/2718967444 shutdown_connections 2024-03-25T18:11:50.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:50.156+0000 7f2858d0e640 1 -- 172.21.15.156:0/2718967444 wait complete. 2024-03-25T18:11:50.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:50.156+0000 7f2858d0e640 1 librados: shutdown 2024-03-25T18:11:50.168 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:56.974 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:57.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.084+0000 7fb409ac9640 10 monclient: get_monmap_and_config 2024-03-25T18:11:57.086 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.084+0000 7fb409ac9640 10 monclient: build_initial_monmap 2024-03-25T18:11:57.086 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.084+0000 7fb409ac9640 10 monclient: monmap: 2024-03-25T18:11:57.086 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:11:57.086 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:57.086 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:11:57.085677+0000 2024-03-25T18:11:57.086 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:11:57.085677+0000 2024-03-25T18:11:57.086 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:57.086 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:57.086 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:11:57.086 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:11:57.086 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:11:57.086 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:57.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.085+0000 7fb409ac9640 1 Processor -- start 2024-03-25T18:11:57.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.085+0000 7fb409ac9640 1 -- start start 2024-03-25T18:11:57.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.085+0000 7fb409ac9640 10 monclient: init 2024-03-25T18:11:57.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.086+0000 7fb409ac9640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:57.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.086+0000 7fb409ac9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:11:57.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.086+0000 7fb409ac9640 10 monclient(hunting): picked mon.noname-b con 0x7fb4040a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:11:57.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.086+0000 7fb409ac9640 10 monclient(hunting): picked mon.noname-c con 0x7fb40413f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:11:57.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.086+0000 7fb409ac9640 10 monclient(hunting): picked mon.noname-a con 0x7fb4041430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:11:57.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.086+0000 7fb409ac9640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb404142520 con 0x7fb4041430c0 2024-03-25T18:11:57.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.086+0000 7fb409ac9640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4040b28a0 con 0x7fb40413f710 2024-03-25T18:11:57.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.086+0000 7fb409ac9640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4040b2a30 con 0x7fb4040a98f0 2024-03-25T18:11:57.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.086+0000 7fb409ac9640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:57.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.086+0000 7fb409ac9640 10 monclient(hunting): authenticate will time out at 7198.397471s 2024-03-25T18:11:57.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.092+0000 7fb403fff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fb4041430c0 0x7fb40414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:37032/0 (socket says 172.21.15.156:37032) 2024-03-25T18:11:57.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.093+0000 7fb403fff640 1 -- 172.21.15.156:0/655801582 learned_addr learned my addr 172.21.15.156:0/655801582 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:11:57.108 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.107+0000 7fb4027fc640 1 -- 172.21.15.156:0/655801582 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2129520838 0 0) 0x7fb404142520 con 0x7fb4041430c0 2024-03-25T18:11:57.108 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.107+0000 7fb4027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:57.108 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.107+0000 7fb4027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:57.108 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.107+0000 7fb4027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:57.108 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.107+0000 7fb4027fc640 10 monclient(hunting): my global_id is 4386 2024-03-25T18:11:57.108 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.107+0000 7fb4027fc640 1 -- 172.21.15.156:0/655801582 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3ec003520 con 0x7fb4041430c0 2024-03-25T18:11:57.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.109+0000 7fb4027fc640 1 -- 172.21.15.156:0/655801582 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1610074644 0 0) 0x7fb4040b2a30 con 0x7fb4040a98f0 2024-03-25T18:11:57.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.109+0000 7fb4027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:57.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.109+0000 7fb4027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:57.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.109+0000 7fb4027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:57.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.109+0000 7fb4027fc640 10 monclient(hunting): my global_id is 4394 2024-03-25T18:11:57.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.109+0000 7fb4027fc640 1 -- 172.21.15.156:0/655801582 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb404142520 con 0x7fb4040a98f0 2024-03-25T18:11:57.117 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.116+0000 7fb4027fc640 1 -- 172.21.15.156:0/655801582 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4113509277 0 0) 0x7fb4040b28a0 con 0x7fb40413f710 2024-03-25T18:11:57.117 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.116+0000 7fb4027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:57.117 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.116+0000 7fb4027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:57.117 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.116+0000 7fb4027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:57.117 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.116+0000 7fb4027fc640 10 monclient(hunting): my global_id is 4384 2024-03-25T18:11:57.117 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.116+0000 7fb4027fc640 1 -- 172.21.15.156:0/655801582 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4040b2a30 con 0x7fb40413f710 2024-03-25T18:11:57.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.120+0000 7fb4027fc640 1 -- 172.21.15.156:0/655801582 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2306261173 0 0) 0x7fb3ec003520 con 0x7fb4041430c0 2024-03-25T18:11:57.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.120+0000 7fb4027fc640 1 -- 172.21.15.156:0/655801582 >> v1:172.21.15.156:6790/0 conn(0x7fb4040a98f0 legacy=0x7fb4040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:57.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.120+0000 7fb4027fc640 1 -- 172.21.15.156:0/655801582 >> v1:172.21.15.194:6789/0 conn(0x7fb40413f710 legacy=0x7fb4041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:57.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.120+0000 7fb4027fc640 10 monclient: _finish_hunting 0 2024-03-25T18:11:57.122 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.120+0000 7fb4027fc640 1 monclient: found mon.noname-a 2024-03-25T18:11:57.122 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.120+0000 7fb4027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:57.122 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.120+0000 7fb4027fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:11:57.122 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.120+0000 7fb4027fc640 1 -- 172.21.15.156:0/655801582 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4040b44c0 con 0x7fb4041430c0 2024-03-25T18:11:57.122 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.120+0000 7fb4027fc640 10 monclient: _finish_auth 0 2024-03-25T18:11:57.122 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.120+0000 7fb4027fc640 10 monclient: _check_auth_tickets 2024-03-25T18:11:57.122 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.120+0000 7fb4027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:57.122 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.120+0000 7fb4027fc640 1 -- 172.21.15.156:0/655801582 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fb3fc001a10 con 0x7fb4041430c0 2024-03-25T18:11:57.122 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.120+0000 7fb4027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:57.122 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.120+0000 7fb4027fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:57.122 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.120+0000 7fb4027fc640 10 monclient: dump: 2024-03-25T18:11:57.122 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:57.122 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:57.122 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:57.123 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:57.123 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:57.123 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:57.123 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:57.123 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:57.123 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:57.123 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:57.123 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.120+0000 7fb409ac9640 5 monclient: authenticate success, global_id 4386 2024-03-25T18:11:57.123 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.120+0000 7fb409ac9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:57.128 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.128+0000 7fb4027fc640 1 -- 172.21.15.156:0/655801582 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3fc001b70 con 0x7fb4041430c0 2024-03-25T18:11:57.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.128+0000 7fb4027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:57.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.128+0000 7fb4027fc640 1 -- 172.21.15.156:0/655801582 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fb3fc001f80 con 0x7fb4041430c0 2024-03-25T18:11:57.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.128+0000 7fb409ac9640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:57.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.128+0000 7fb409ac9640 4 set_mon_vals no callback set 2024-03-25T18:11:57.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.128+0000 7fb4027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:57.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.128+0000 7fb4027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:57.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.128+0000 7fb4027fc640 10 monclient: dump: 2024-03-25T18:11:57.129 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:57.129 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:57.129 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:57.129 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:57.129 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:57.129 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:57.129 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:57.130 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:57.130 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:57.130 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:57.131 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.128+0000 7fb409ac9640 10 monclient: shutdown 2024-03-25T18:11:57.131 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.128+0000 7fb409ac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:57.131 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.128+0000 7fb409ac9640 1 -- 172.21.15.156:0/655801582 >> v1:172.21.15.156:6789/0 conn(0x7fb4041430c0 legacy=0x7fb40414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:57.131 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.128+0000 7fb409ac9640 1 -- 172.21.15.156:0/655801582 shutdown_connections 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.128+0000 7fb409ac9640 1 -- 172.21.15.156:0/655801582 >> 172.21.15.156:0/655801582 conn(0x7fb4040b7ea0 msgr2=0x7fb4040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.129+0000 7fb409ac9640 1 -- 172.21.15.156:0/655801582 shutdown_connections 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.129+0000 7fb409ac9640 1 -- 172.21.15.156:0/655801582 wait complete. 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.129+0000 7fb409ac9640 10 monclient: build_initial_monmap 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.129+0000 7fb409ac9640 10 monclient: monmap: 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:11:57.132 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:57.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.130+0000 7fb409ac9640 1 librados: starting msgr at 2024-03-25T18:11:57.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.130+0000 7fb409ac9640 1 librados: starting objecter 2024-03-25T18:11:57.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.130+0000 7fb409ac9640 1 Processor -- start 2024-03-25T18:11:57.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.130+0000 7fb409ac9640 1 -- start start 2024-03-25T18:11:57.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.130+0000 7fb409ac9640 1 librados: setting wanted keys 2024-03-25T18:11:57.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.130+0000 7fb409ac9640 1 librados: calling monclient init 2024-03-25T18:11:57.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.130+0000 7fb409ac9640 10 monclient: init 2024-03-25T18:11:57.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.131+0000 7fb409ac9640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:57.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.131+0000 7fb409ac9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:11:57.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.131+0000 7fb409ac9640 10 monclient(hunting): picked mon.noname-b con 0x7fb4040a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:11:57.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.131+0000 7fb409ac9640 10 monclient(hunting): picked mon.noname-a con 0x7fb40413f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:11:57.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.131+0000 7fb409ac9640 10 monclient(hunting): picked mon.noname-c con 0x7fb4041430c0 addr v1:172.21.15.156:6790/0 2024-03-25T18:11:57.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.131+0000 7fb409ac9640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb404136c50 con 0x7fb40413f710 2024-03-25T18:11:57.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.131+0000 7fb409ac9640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb404136ec0 con 0x7fb4040a98f0 2024-03-25T18:11:57.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.131+0000 7fb409ac9640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4041317f0 con 0x7fb4041430c0 2024-03-25T18:11:57.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.131+0000 7fb409ac9640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:57.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.131+0000 7fb409ac9640 10 monclient(hunting): authenticate will time out at 7198.442548s 2024-03-25T18:11:57.142 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.141+0000 7fb403fff640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7fb4041430c0 0x7fb4041db380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:57972/0 (socket says 172.21.15.156:57972) 2024-03-25T18:11:57.142 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.141+0000 7fb403fff640 1 -- 172.21.15.156:0/278078882 learned_addr learned my addr 172.21.15.156:0/278078882 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:11:57.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.143+0000 7fb3ebfff640 10 client.?.objecter ms_handle_connect 0x7fb4041430c0 2024-03-25T18:11:57.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.143+0000 7fb3ebfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:57.148 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.147+0000 7fb3ebfff640 10 client.?.objecter ms_handle_connect 0x7fb40413f710 2024-03-25T18:11:57.148 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.147+0000 7fb3ebfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:57.151 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.150+0000 7fb3ebfff640 1 -- 172.21.15.156:0/278078882 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2587101823 0 0) 0x7fb404136c50 con 0x7fb40413f710 2024-03-25T18:11:57.151 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.150+0000 7fb3ebfff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:57.151 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.150+0000 7fb3ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:57.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.150+0000 7fb3ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:57.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.150+0000 7fb3ebfff640 10 monclient(hunting): my global_id is 4389 2024-03-25T18:11:57.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.150+0000 7fb3ebfff640 1 -- 172.21.15.156:0/278078882 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3f0004eb0 con 0x7fb40413f710 2024-03-25T18:11:57.161 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.160+0000 7fb3ebfff640 1 -- 172.21.15.156:0/278078882 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1762200894 0 0) 0x7fb4041317f0 con 0x7fb4041430c0 2024-03-25T18:11:57.161 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.160+0000 7fb3ebfff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:57.161 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.160+0000 7fb3ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:57.161 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.160+0000 7fb3ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:57.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.160+0000 7fb3ebfff640 10 monclient(hunting): my global_id is 4397 2024-03-25T18:11:57.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.160+0000 7fb3ebfff640 1 -- 172.21.15.156:0/278078882 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb404136c50 con 0x7fb4041430c0 2024-03-25T18:11:57.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.160+0000 7fb3ebfff640 10 client.?.objecter ms_handle_connect 0x7fb4040a98f0 2024-03-25T18:11:57.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.160+0000 7fb3ebfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:57.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.160+0000 7fb3ebfff640 1 -- 172.21.15.156:0/278078882 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1784167801 0 0) 0x7fb3f0004eb0 con 0x7fb40413f710 2024-03-25T18:11:57.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.160+0000 7fb3ebfff640 1 -- 172.21.15.156:0/278078882 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4041317f0 con 0x7fb40413f710 2024-03-25T18:11:57.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.160+0000 7fb3ebfff640 1 -- 172.21.15.156:0/278078882 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fb3f80018f0 con 0x7fb40413f710 2024-03-25T18:11:57.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.160+0000 7fb3ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:57.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.160+0000 7fb3ebfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:57.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.160+0000 7fb3ebfff640 10 monclient(hunting): dump: 2024-03-25T18:11:57.162 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:57.162 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:57.163 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:57.163 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:57.163 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:57.163 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:57.163 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:57.163 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:57.163 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:57.163 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:57.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.160+0000 7fb3ebfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:57.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.165+0000 7fb3ebfff640 1 -- 172.21.15.156:0/278078882 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3338528338 0 0) 0x7fb4041317f0 con 0x7fb40413f710 2024-03-25T18:11:57.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.165+0000 7fb3ebfff640 1 -- 172.21.15.156:0/278078882 >> v1:172.21.15.156:6790/0 conn(0x7fb4041430c0 legacy=0x7fb4041db380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:57.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.165+0000 7fb3ebfff640 1 -- 172.21.15.156:0/278078882 >> v1:172.21.15.194:6789/0 conn(0x7fb4040a98f0 legacy=0x7fb4041310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:57.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.165+0000 7fb3ebfff640 10 monclient: _finish_hunting 0 2024-03-25T18:11:57.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.165+0000 7fb3ebfff640 1 monclient: found mon.a 2024-03-25T18:11:57.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.165+0000 7fb3ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:57.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.165+0000 7fb3ebfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:11:57.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.165+0000 7fb3ebfff640 1 -- 172.21.15.156:0/278078882 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb404131980 con 0x7fb40413f710 2024-03-25T18:11:57.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.165+0000 7fb3ebfff640 10 monclient: _finish_auth 0 2024-03-25T18:11:57.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.165+0000 7fb3ebfff640 10 monclient: _check_auth_tickets 2024-03-25T18:11:57.168 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.165+0000 7fb3ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:57.168 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.165+0000 7fb409ac9640 5 monclient: authenticate success, global_id 4389 2024-03-25T18:11:57.168 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.165+0000 7fb409ac9640 10 monclient: _renew_subs 2024-03-25T18:11:57.168 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.165+0000 7fb409ac9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:11:57.168 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.165+0000 7fb409ac9640 1 -- 172.21.15.156:0/278078882 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb404135070 con 0x7fb40413f710 2024-03-25T18:11:57.168 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.166+0000 7fb409ac9640 10 client.4389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:57.168 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.166+0000 7fb409ac9640 10 monclient: _renew_subs 2024-03-25T18:11:57.168 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.166+0000 7fb409ac9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:11:57.168 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.166+0000 7fb409ac9640 1 -- 172.21.15.156:0/278078882 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4040b7870 con 0x7fb40413f710 2024-03-25T18:11:57.168 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.166+0000 7fb409ac9640 1 librados: init done 2024-03-25T18:11:57.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.167+0000 7fb409ac9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:57.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.167+0000 7fb409ac9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:57.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.167+0000 7fb409ac9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:11:57.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.167+0000 7fb409ac9640 1 -- 172.21.15.156:0/278078882 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb404135c10 con 0x7fb40413f710 2024-03-25T18:11:57.474 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.473+0000 7fb3ebfff640 1 -- 172.21.15.156:0/278078882 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3f8003610 con 0x7fb40413f710 2024-03-25T18:11:57.474 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.473+0000 7fb3ebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:57.474 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.473+0000 7fb3ebfff640 1 -- 172.21.15.156:0/278078882 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fb3f80039a0 con 0x7fb40413f710 2024-03-25T18:11:57.474 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.473+0000 7fb3ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:57.474 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.473+0000 7fb3ebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:57.474 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.473+0000 7fb3ebfff640 10 monclient: dump: 2024-03-25T18:11:57.475 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:11:57.475 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:11:57.475 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:57.475 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:11:57.475 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:57.475 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:11:57.475 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:11:57.475 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:11:57.475 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:11:57.475 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:11:57.475 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.473+0000 7fb401ffb640 4 set_mon_vals no callback set 2024-03-25T18:11:57.476 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.473+0000 7fb3ebfff640 1 -- 172.21.15.156:0/278078882 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7fb3f8003bc0 con 0x7fb40413f710 2024-03-25T18:11:57.476 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.474+0000 7fb4037fe640 10 monclient: get_auth_request con 0x7fb3f006dd10 auth_method 0 2024-03-25T18:11:57.476 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.474+0000 7fb3ebfff640 10 client.4389.objecter ms_handle_connect 0x7fb3f006dd10 2024-03-25T18:11:57.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.480+0000 7fb3ebfff640 1 -- 172.21.15.156:0/278078882 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7fb3f8053580 con 0x7fb40413f710 2024-03-25T18:11:57.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.480+0000 7fb3ebfff640 10 client.4389.objecter ms_dispatch 0x7fb404000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:57.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.480+0000 7fb3ebfff640 3 client.4389.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:57.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.480+0000 7fb3ebfff640 3 client.4389.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:57.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.480+0000 7fb3ebfff640 20 client.4389.objecter dump_active .. 0 homeless 2024-03-25T18:11:57.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.480+0000 7fb3ebfff640 1 -- 172.21.15.156:0/278078882 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb3f804f090 con 0x7fb40413f710 2024-03-25T18:11:57.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.480+0000 7fb3ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:57.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.480+0000 7fb3ebfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:57.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.728+0000 7fb409ac9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:57.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.728+0000 7fb409ac9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:57.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.728+0000 7fb409ac9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:11:57.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.728+0000 7fb409ac9640 1 -- 172.21.15.156:0/278078882 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4040627b0 con 0x7fb40413f710 2024-03-25T18:11:57.761 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.760+0000 7fb3ebfff640 1 -- 172.21.15.156:0/278078882 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7fb3f8050fc0 con 0x7fb40413f710 2024-03-25T18:11:57.761 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.760+0000 7fb3ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:57.761 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.760+0000 7fb3ebfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:57.761 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:11:57.761 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:57.764 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.762+0000 7fb409ac9640 10 librados: watch_flush enter 2024-03-25T18:11:57.764 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.762+0000 7fb409ac9640 10 librados: watch_flush exit 2024-03-25T18:11:57.764 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.762+0000 7fb409ac9640 20 client.4389.objecter shutdown clearing up homeless session... 2024-03-25T18:11:57.764 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.762+0000 7fb409ac9640 10 client.4389.objecter successfully canceled tick 2024-03-25T18:11:57.764 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.762+0000 7fb409ac9640 1 -- 172.21.15.156:0/278078882 >> v1:172.21.15.194:6800/2946427713 conn(0x7fb3f006dd10 legacy=0x7fb3f00701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:57.764 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.762+0000 7fb409ac9640 10 monclient: shutdown 2024-03-25T18:11:57.764 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.762+0000 7fb409ac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:57.764 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.762+0000 7fb409ac9640 1 -- 172.21.15.156:0/278078882 >> v1:172.21.15.156:6789/0 conn(0x7fb40413f710 legacy=0x7fb4041dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:57.764 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.762+0000 7fb409ac9640 1 -- 172.21.15.156:0/278078882 shutdown_connections 2024-03-25T18:11:57.764 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.762+0000 7fb409ac9640 1 -- 172.21.15.156:0/278078882 >> 172.21.15.156:0/278078882 conn(0x7fb4040b7ea0 msgr2=0x7fb40413d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:57.764 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.762+0000 7fb409ac9640 1 -- 172.21.15.156:0/278078882 shutdown_connections 2024-03-25T18:11:57.765 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.763+0000 7fb409ac9640 1 -- 172.21.15.156:0/278078882 wait complete. 2024-03-25T18:11:57.765 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:11:57.763+0000 7fb409ac9640 1 librados: shutdown 2024-03-25T18:11:57.776 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:03.801 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:04:09.792 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:03.806 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:04:09.797 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:03.847 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:04:09.838 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:03.849 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:04:09.816 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:03.859 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:04:09.826 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:03.871 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:04:09.861 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:03.875 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:04:09.866 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:03.889 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:04:09.856 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:03.905 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:04:09.895 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:04.583 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:04.698 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.697+0000 7f81f7fcf640 10 monclient: get_monmap_and_config 2024-03-25T18:12:04.698 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.697+0000 7f81f7fcf640 10 monclient: build_initial_monmap 2024-03-25T18:12:04.698 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.697+0000 7f81f7fcf640 10 monclient: monmap: 2024-03-25T18:12:04.698 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:12:04.698 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:04.698 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:12:04.698295+0000 2024-03-25T18:12:04.698 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:12:04.698295+0000 2024-03-25T18:12:04.698 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:04.699 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:04.699 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:12:04.699 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:12:04.699 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:12:04.699 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:04.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.698+0000 7f81f7fcf640 1 Processor -- start 2024-03-25T18:12:04.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.698+0000 7f81f7fcf640 1 -- start start 2024-03-25T18:12:04.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.698+0000 7f81f7fcf640 10 monclient: init 2024-03-25T18:12:04.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.698+0000 7f81f7fcf640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:04.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.698+0000 7f81f7fcf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:12:04.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.698+0000 7f81f7fcf640 10 monclient(hunting): picked mon.noname-a con 0x7f81f00a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:12:04.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.698+0000 7f81f7fcf640 10 monclient(hunting): picked mon.noname-c con 0x7f81f013f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:12:04.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.699+0000 7f81f7fcf640 10 monclient(hunting): picked mon.noname-b con 0x7f81f01430c0 addr v1:172.21.15.156:6790/0 2024-03-25T18:12:04.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.699+0000 7f81f7fcf640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81f0142520 con 0x7f81f00a98f0 2024-03-25T18:12:04.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.699+0000 7f81f7fcf640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81f00b28a0 con 0x7f81f013f710 2024-03-25T18:12:04.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.699+0000 7f81f7fcf640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81f00b2a30 con 0x7f81f01430c0 2024-03-25T18:12:04.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.699+0000 7f81f7fcf640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:04.701 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.699+0000 7f81f7fcf640 10 monclient(hunting): authenticate will time out at 7206.009834s 2024-03-25T18:12:04.714 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.713+0000 7f81f556a640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f81f013f710 0x7f81f01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:34438/0 (socket says 172.21.15.156:34438) 2024-03-25T18:12:04.714 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.713+0000 7f81f556a640 1 -- 172.21.15.156:0/3094800488 learned_addr learned my addr 172.21.15.156:0/3094800488 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:12:04.724 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.723+0000 7f81f4d69640 1 -- 172.21.15.156:0/3094800488 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404921957 0 0) 0x7f81f0142520 con 0x7f81f00a98f0 2024-03-25T18:12:04.724 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.723+0000 7f81f4d69640 10 monclient(hunting): _negotiate 2024-03-25T18:12:04.724 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.723+0000 7f81f4d69640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:04.724 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.723+0000 7f81f4d69640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:04.724 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.723+0000 7f81f4d69640 10 monclient(hunting): my global_id is 4392 2024-03-25T18:12:04.724 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.723+0000 7f81f4d69640 1 -- 172.21.15.156:0/3094800488 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81dc003520 con 0x7f81f00a98f0 2024-03-25T18:12:04.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.728+0000 7f81f4d69640 1 -- 172.21.15.156:0/3094800488 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2393354497 0 0) 0x7f81f00b28a0 con 0x7f81f013f710 2024-03-25T18:12:04.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.728+0000 7f81f4d69640 10 monclient(hunting): _negotiate 2024-03-25T18:12:04.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.728+0000 7f81f4d69640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:04.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.728+0000 7f81f4d69640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:04.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.728+0000 7f81f4d69640 10 monclient(hunting): my global_id is 4390 2024-03-25T18:12:04.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.728+0000 7f81f4d69640 1 -- 172.21.15.156:0/3094800488 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81f0142520 con 0x7f81f013f710 2024-03-25T18:12:04.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.729+0000 7f81f4d69640 1 -- 172.21.15.156:0/3094800488 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871479099 0 0) 0x7f81f00b2a30 con 0x7f81f01430c0 2024-03-25T18:12:04.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.729+0000 7f81f4d69640 10 monclient(hunting): _negotiate 2024-03-25T18:12:04.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.729+0000 7f81f4d69640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:04.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.729+0000 7f81f4d69640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:04.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.729+0000 7f81f4d69640 10 monclient(hunting): my global_id is 4400 2024-03-25T18:12:04.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.729+0000 7f81f4d69640 1 -- 172.21.15.156:0/3094800488 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81f00b28a0 con 0x7f81f01430c0 2024-03-25T18:12:04.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.732+0000 7f81f4d69640 1 -- 172.21.15.156:0/3094800488 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2324329269 0 0) 0x7f81dc003520 con 0x7f81f00a98f0 2024-03-25T18:12:04.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.732+0000 7f81f4d69640 1 -- 172.21.15.156:0/3094800488 >> v1:172.21.15.156:6790/0 conn(0x7f81f01430c0 legacy=0x7f81f014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:04.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.732+0000 7f81f4d69640 1 -- 172.21.15.156:0/3094800488 >> v1:172.21.15.194:6789/0 conn(0x7f81f013f710 legacy=0x7f81f01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:04.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.732+0000 7f81f4d69640 10 monclient: _finish_hunting 0 2024-03-25T18:12:04.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.732+0000 7f81f4d69640 1 monclient: found mon.noname-a 2024-03-25T18:12:04.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.732+0000 7f81f4d69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:04.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.732+0000 7f81f4d69640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:12:04.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.732+0000 7f81f4d69640 1 -- 172.21.15.156:0/3094800488 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f00b44c0 con 0x7f81f00a98f0 2024-03-25T18:12:04.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.732+0000 7f81f4d69640 10 monclient: _finish_auth 0 2024-03-25T18:12:04.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.732+0000 7f81f4d69640 10 monclient: _check_auth_tickets 2024-03-25T18:12:04.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.732+0000 7f81f4d69640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:04.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.732+0000 7f81f4d69640 1 -- 172.21.15.156:0/3094800488 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f81ec001690 con 0x7f81f00a98f0 2024-03-25T18:12:04.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.732+0000 7f81f4d69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:04.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.732+0000 7f81f4d69640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:04.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.732+0000 7f81f4d69640 10 monclient: dump: 2024-03-25T18:12:04.734 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:04.734 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:04.734 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:04.735 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:04.735 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:04.735 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:04.735 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:04.735 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:04.735 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:04.735 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:04.735 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.732+0000 7f81f7fcf640 5 monclient: authenticate success, global_id 4392 2024-03-25T18:12:04.735 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.732+0000 7f81f7fcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:04.753 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.752+0000 7f81f4d69640 1 -- 172.21.15.156:0/3094800488 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81ec0017f0 con 0x7f81f00a98f0 2024-03-25T18:12:04.753 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.752+0000 7f81f4d69640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:04.753 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.752+0000 7f81f4d69640 1 -- 172.21.15.156:0/3094800488 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f81ec001c00 con 0x7f81f00a98f0 2024-03-25T18:12:04.753 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.752+0000 7f81f7fcf640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:04.753 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.752+0000 7f81f7fcf640 4 set_mon_vals no callback set 2024-03-25T18:12:04.754 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.752+0000 7f81f4d69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:04.754 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.752+0000 7f81f4d69640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:04.754 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.752+0000 7f81f4d69640 10 monclient: dump: 2024-03-25T18:12:04.754 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:04.754 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:04.754 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:04.754 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:04.754 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:04.754 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:04.754 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:04.754 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:04.754 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:04.754 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:04.757 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.753+0000 7f81f7fcf640 10 monclient: shutdown 2024-03-25T18:12:04.757 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.753+0000 7f81f7fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:04.757 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.753+0000 7f81f7fcf640 1 -- 172.21.15.156:0/3094800488 >> v1:172.21.15.156:6789/0 conn(0x7f81f00a98f0 legacy=0x7f81f00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:04.757 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.753+0000 7f81f7fcf640 1 -- 172.21.15.156:0/3094800488 shutdown_connections 2024-03-25T18:12:04.757 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.753+0000 7f81f7fcf640 1 -- 172.21.15.156:0/3094800488 >> 172.21.15.156:0/3094800488 conn(0x7f81f00b7ea0 msgr2=0x7f81f00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:04.757 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.754+0000 7f81f7fcf640 1 -- 172.21.15.156:0/3094800488 shutdown_connections 2024-03-25T18:12:04.757 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.754+0000 7f81f7fcf640 1 -- 172.21.15.156:0/3094800488 wait complete. 2024-03-25T18:12:04.757 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.754+0000 7f81f7fcf640 10 monclient: build_initial_monmap 2024-03-25T18:12:04.757 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.754+0000 7f81f7fcf640 10 monclient: monmap: 2024-03-25T18:12:04.758 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:12:04.758 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:04.758 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:12:04.758 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:12:04.758 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:04.758 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:12:04.758 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:12:04.758 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:12:04.758 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:12:04.758 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:04.758 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.755+0000 7f81f7fcf640 1 librados: starting msgr at 2024-03-25T18:12:04.758 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.755+0000 7f81f7fcf640 1 librados: starting objecter 2024-03-25T18:12:04.758 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.755+0000 7f81f7fcf640 1 Processor -- start 2024-03-25T18:12:04.758 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.755+0000 7f81f7fcf640 1 -- start start 2024-03-25T18:12:04.758 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.755+0000 7f81f7fcf640 1 librados: setting wanted keys 2024-03-25T18:12:04.759 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.755+0000 7f81f7fcf640 1 librados: calling monclient init 2024-03-25T18:12:04.759 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.755+0000 7f81f7fcf640 10 monclient: init 2024-03-25T18:12:04.759 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.755+0000 7f81f7fcf640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:04.759 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.755+0000 7f81f7fcf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:12:04.759 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.755+0000 7f81f7fcf640 10 monclient(hunting): picked mon.noname-b con 0x7f81f00a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:12:04.759 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.756+0000 7f81f7fcf640 10 monclient(hunting): picked mon.noname-c con 0x7f81f013f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:12:04.759 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.756+0000 7f81f7fcf640 10 monclient(hunting): picked mon.noname-a con 0x7f81f01430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:12:04.759 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.756+0000 7f81f7fcf640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81f00abd30 con 0x7f81f01430c0 2024-03-25T18:12:04.759 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.756+0000 7f81f7fcf640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81f01d77a0 con 0x7f81f00a98f0 2024-03-25T18:12:04.759 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.756+0000 7f81f7fcf640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81f01d51f0 con 0x7f81f013f710 2024-03-25T18:12:04.759 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.756+0000 7f81f7fcf640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:04.759 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.756+0000 7f81f7fcf640 10 monclient(hunting): authenticate will time out at 7206.066994s 2024-03-25T18:12:04.769 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.768+0000 7f81f5d6b640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f81f00a98f0 0x7f81f01d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:34482/0 (socket says 172.21.15.156:34482) 2024-03-25T18:12:04.769 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.768+0000 7f81f5d6b640 1 -- 172.21.15.156:0/1760385732 learned_addr learned my addr 172.21.15.156:0/1760385732 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:12:04.771 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.770+0000 7f81e67fc640 10 client.?.objecter ms_handle_connect 0x7f81f00a98f0 2024-03-25T18:12:04.771 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.770+0000 7f81e67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:04.774 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.774+0000 7f81e67fc640 10 client.?.objecter ms_handle_connect 0x7f81f013f710 2024-03-25T18:12:04.774 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.774+0000 7f81e67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:04.785 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.784+0000 7f81e67fc640 10 client.?.objecter ms_handle_connect 0x7f81f01430c0 2024-03-25T18:12:04.785 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.784+0000 7f81e67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:04.855 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.854+0000 7f81e67fc640 1 -- 172.21.15.156:0/1760385732 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 77001510 0 0) 0x7f81f00abd30 con 0x7f81f01430c0 2024-03-25T18:12:04.856 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.854+0000 7f81e67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:04.856 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.854+0000 7f81e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:04.856 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.854+0000 7f81e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:04.856 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.854+0000 7f81e67fc640 10 monclient(hunting): my global_id is 4395 2024-03-25T18:12:04.856 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.854+0000 7f81e67fc640 1 -- 172.21.15.156:0/1760385732 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81d4004eb0 con 0x7f81f01430c0 2024-03-25T18:12:04.862 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.861+0000 7f81e67fc640 1 -- 172.21.15.156:0/1760385732 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918586100 0 0) 0x7f81f01d77a0 con 0x7f81f00a98f0 2024-03-25T18:12:04.862 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.861+0000 7f81e67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:04.862 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.861+0000 7f81e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:04.862 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.861+0000 7f81e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:04.862 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.861+0000 7f81e67fc640 10 monclient(hunting): my global_id is 4393 2024-03-25T18:12:04.862 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.861+0000 7f81e67fc640 1 -- 172.21.15.156:0/1760385732 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81f00abd30 con 0x7f81f00a98f0 2024-03-25T18:12:04.871 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.870+0000 7f81e67fc640 1 -- 172.21.15.156:0/1760385732 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1660527922 0 0) 0x7f81f01d51f0 con 0x7f81f013f710 2024-03-25T18:12:04.871 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.870+0000 7f81e67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:04.871 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.870+0000 7f81e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:04.871 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.870+0000 7f81e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:04.871 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.870+0000 7f81e67fc640 10 monclient(hunting): my global_id is 4403 2024-03-25T18:12:04.871 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.870+0000 7f81e67fc640 1 -- 172.21.15.156:0/1760385732 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81f01d77a0 con 0x7f81f013f710 2024-03-25T18:12:04.894 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.893+0000 7f81e67fc640 1 -- 172.21.15.156:0/1760385732 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1623162672 0 0) 0x7f81d4004eb0 con 0x7f81f01430c0 2024-03-25T18:12:04.894 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.893+0000 7f81e67fc640 1 -- 172.21.15.156:0/1760385732 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81f01d51f0 con 0x7f81f01430c0 2024-03-25T18:12:04.894 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.893+0000 7f81e67fc640 1 -- 172.21.15.156:0/1760385732 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f81e8001bd0 con 0x7f81f01430c0 2024-03-25T18:12:04.894 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.893+0000 7f81e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:04.894 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.893+0000 7f81e67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:04.894 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.893+0000 7f81e67fc640 10 monclient(hunting): dump: 2024-03-25T18:12:04.894 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:04.894 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:04.894 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:04.894 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:04.894 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:04.895 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:04.895 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:04.895 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:04.895 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:04.895 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:04.895 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.893+0000 7f81e67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:04.958 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.957+0000 7f81e67fc640 1 -- 172.21.15.156:0/1760385732 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3695023812 0 0) 0x7f81f00abd30 con 0x7f81f00a98f0 2024-03-25T18:12:04.959 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.957+0000 7f81e67fc640 1 -- 172.21.15.156:0/1760385732 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81d4004eb0 con 0x7f81f00a98f0 2024-03-25T18:12:04.959 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.957+0000 7f81e67fc640 1 -- 172.21.15.156:0/1760385732 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f81ec001b40 con 0x7f81f00a98f0 2024-03-25T18:12:04.959 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.957+0000 7f81e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:04.959 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.957+0000 7f81e67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:04.959 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.957+0000 7f81e67fc640 10 monclient(hunting): dump: 2024-03-25T18:12:04.959 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:04.959 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:04.959 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:04.959 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:04.959 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:04.959 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:04.959 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:04.959 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:04.959 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:04.959 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:04.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.959+0000 7f81e67fc640 1 -- 172.21.15.156:0/1760385732 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3360916344 0 0) 0x7f81f01d51f0 con 0x7f81f01430c0 2024-03-25T18:12:04.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.960+0000 7f81e67fc640 1 -- 172.21.15.156:0/1760385732 >> v1:172.21.15.156:6790/0 conn(0x7f81f013f710 legacy=0x7f81f0138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:04.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.960+0000 7f81e67fc640 1 -- 172.21.15.156:0/1760385732 >> v1:172.21.15.194:6789/0 conn(0x7f81f00a98f0 legacy=0x7f81f01d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:04.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.960+0000 7f81e67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:12:04.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.960+0000 7f81e67fc640 1 monclient: found mon.a 2024-03-25T18:12:04.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.960+0000 7f81e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:04.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.960+0000 7f81e67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:12:04.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.960+0000 7f81e67fc640 1 -- 172.21.15.156:0/1760385732 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f01d83d0 con 0x7f81f01430c0 2024-03-25T18:12:04.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.960+0000 7f81e67fc640 10 monclient: _finish_auth 0 2024-03-25T18:12:04.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.960+0000 7f81e67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:12:04.963 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.960+0000 7f81e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:04.963 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.960+0000 7f81f7fcf640 5 monclient: authenticate success, global_id 4395 2024-03-25T18:12:04.963 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.960+0000 7f81f7fcf640 10 monclient: _renew_subs 2024-03-25T18:12:04.963 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.960+0000 7f81f7fcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:12:04.963 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.960+0000 7f81f7fcf640 1 -- 172.21.15.156:0/1760385732 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81f0138a50 con 0x7f81f01430c0 2024-03-25T18:12:04.963 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.960+0000 7f81f7fcf640 10 client.4395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:04.963 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.960+0000 7f81f7fcf640 10 monclient: _renew_subs 2024-03-25T18:12:04.963 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.960+0000 7f81f7fcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:12:04.963 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.960+0000 7f81f7fcf640 1 -- 172.21.15.156:0/1760385732 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81f01e35c0 con 0x7f81f01430c0 2024-03-25T18:12:04.963 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.960+0000 7f81f7fcf640 1 librados: init done 2024-03-25T18:12:04.963 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.961+0000 7f81f7fcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:04.963 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.961+0000 7f81f7fcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:04.963 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.961+0000 7f81f7fcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:12:04.963 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:04.961+0000 7f81f7fcf640 1 -- 172.21.15.156:0/1760385732 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81f00627b0 con 0x7f81f01430c0 2024-03-25T18:12:05.276 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.274+0000 7f81e67fc640 1 -- 172.21.15.156:0/1760385732 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81e8002060 con 0x7f81f01430c0 2024-03-25T18:12:05.276 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.274+0000 7f81e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:05.276 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.274+0000 7f81e67fc640 1 -- 172.21.15.156:0/1760385732 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f81e8003b30 con 0x7f81f01430c0 2024-03-25T18:12:05.276 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.274+0000 7f81e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:05.276 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.274+0000 7f81e67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:05.276 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.274+0000 7f81e67fc640 10 monclient: dump: 2024-03-25T18:12:05.276 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:05.276 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:05.276 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:05.276 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:05.277 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:05.277 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:05.277 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:05.277 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:05.277 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:05.277 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:05.277 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.274+0000 7f81e7fff640 4 set_mon_vals no callback set 2024-03-25T18:12:05.278 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.275+0000 7f81e67fc640 1 -- 172.21.15.156:0/1760385732 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f81e8019b00 con 0x7f81f01430c0 2024-03-25T18:12:05.278 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.276+0000 7f81f5d6b640 10 monclient: get_auth_request con 0x7f81d406df20 auth_method 0 2024-03-25T18:12:05.278 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.276+0000 7f81e67fc640 10 client.4395.objecter ms_handle_connect 0x7f81d406df20 2024-03-25T18:12:05.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.281+0000 7f81e67fc640 1 -- 172.21.15.156:0/1760385732 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f81e8053860 con 0x7f81f01430c0 2024-03-25T18:12:05.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.281+0000 7f81e67fc640 10 client.4395.objecter ms_dispatch 0x7f81f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:05.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.281+0000 7f81e67fc640 3 client.4395.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:05.283 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.281+0000 7f81e67fc640 3 client.4395.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:05.283 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.281+0000 7f81e67fc640 20 client.4395.objecter dump_active .. 0 homeless 2024-03-25T18:12:05.283 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.281+0000 7f81e67fc640 1 -- 172.21.15.156:0/1760385732 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f81e804f370 con 0x7f81f01430c0 2024-03-25T18:12:05.283 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.281+0000 7f81e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:05.283 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.281+0000 7f81e67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:05.525 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.524+0000 7f81f7fcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:05.525 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.524+0000 7f81f7fcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:05.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.524+0000 7f81f7fcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:12:05.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.524+0000 7f81f7fcf640 1 -- 172.21.15.156:0/1760385732 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81f0074cc0 con 0x7f81f01430c0 2024-03-25T18:12:05.575 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.574+0000 7f81e67fc640 1 -- 172.21.15.156:0/1760385732 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f81e80512a0 con 0x7f81f01430c0 2024-03-25T18:12:05.575 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.574+0000 7f81e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:05.575 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.574+0000 7f81e67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:05.575 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:12:05.576 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:05.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.577+0000 7f81f7fcf640 10 librados: watch_flush enter 2024-03-25T18:12:05.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.577+0000 7f81f7fcf640 10 librados: watch_flush exit 2024-03-25T18:12:05.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.577+0000 7f81f7fcf640 20 client.4395.objecter shutdown clearing up homeless session... 2024-03-25T18:12:05.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.577+0000 7f81f7fcf640 10 client.4395.objecter successfully canceled tick 2024-03-25T18:12:05.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.577+0000 7f81f7fcf640 1 -- 172.21.15.156:0/1760385732 >> v1:172.21.15.194:6800/2946427713 conn(0x7f81d406df20 legacy=0x7f81d40703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:05.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.577+0000 7f81f7fcf640 10 monclient: shutdown 2024-03-25T18:12:05.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.577+0000 7f81f7fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:05.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.577+0000 7f81f7fcf640 1 -- 172.21.15.156:0/1760385732 >> v1:172.21.15.156:6789/0 conn(0x7f81f01430c0 legacy=0x7f81f01e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:05.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.577+0000 7f81f7fcf640 1 -- 172.21.15.156:0/1760385732 shutdown_connections 2024-03-25T18:12:05.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.577+0000 7f81f7fcf640 1 -- 172.21.15.156:0/1760385732 >> 172.21.15.156:0/1760385732 conn(0x7f81f00b7ea0 msgr2=0x7f81f0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:05.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.577+0000 7f81f7fcf640 1 -- 172.21.15.156:0/1760385732 shutdown_connections 2024-03-25T18:12:05.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.577+0000 7f81f7fcf640 1 -- 172.21.15.156:0/1760385732 wait complete. 2024-03-25T18:12:05.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:05.577+0000 7f81f7fcf640 1 librados: shutdown 2024-03-25T18:12:05.591 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:12.398 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:12.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.508+0000 7f8ae1e55640 10 monclient: get_monmap_and_config 2024-03-25T18:12:12.510 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.508+0000 7f8ae1e55640 10 monclient: build_initial_monmap 2024-03-25T18:12:12.510 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.509+0000 7f8ae1e55640 10 monclient: monmap: 2024-03-25T18:12:12.510 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:12:12.510 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:12.510 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:12:12.510022+0000 2024-03-25T18:12:12.510 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:12:12.510022+0000 2024-03-25T18:12:12.510 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:12.510 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:12.510 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:12:12.510 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:12:12.510 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:12:12.510 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:12.511 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.510+0000 7f8ae1e55640 1 Processor -- start 2024-03-25T18:12:12.511 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.510+0000 7f8ae1e55640 1 -- start start 2024-03-25T18:12:12.511 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.510+0000 7f8ae1e55640 10 monclient: init 2024-03-25T18:12:12.512 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.511+0000 7f8ae1e55640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:12.512 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.511+0000 7f8ae1e55640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:12:12.512 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.511+0000 7f8ae1e55640 10 monclient(hunting): picked mon.noname-a con 0x7f8adc0a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:12:12.512 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.511+0000 7f8ae1e55640 10 monclient(hunting): picked mon.noname-b con 0x7f8adc13f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:12:12.512 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.511+0000 7f8ae1e55640 10 monclient(hunting): picked mon.noname-c con 0x7f8adc1430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:12:12.512 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.511+0000 7f8ae1e55640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8adc142520 con 0x7f8adc0a98f0 2024-03-25T18:12:12.512 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.511+0000 7f8ae1e55640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8adc0b28a0 con 0x7f8adc1430c0 2024-03-25T18:12:12.512 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.511+0000 7f8ae1e55640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8adc0b2a30 con 0x7f8adc13f710 2024-03-25T18:12:12.512 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.511+0000 7f8ae1e55640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:12.512 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.511+0000 7f8ae1e55640 10 monclient(hunting): authenticate will time out at 7213.821961s 2024-03-25T18:12:12.520 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.519+0000 7f8adb7fe640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f8adc0a98f0 0x7f8adc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:48806/0 (socket says 172.21.15.156:48806) 2024-03-25T18:12:12.520 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.519+0000 7f8adb7fe640 1 -- 172.21.15.156:0/2990677225 learned_addr learned my addr 172.21.15.156:0/2990677225 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:12:12.533 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.533+0000 7f8ada7fc640 1 -- 172.21.15.156:0/2990677225 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1288832441 0 0) 0x7f8adc0b28a0 con 0x7f8adc1430c0 2024-03-25T18:12:12.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.533+0000 7f8ada7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:12.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.533+0000 7f8ada7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:12.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.533+0000 7f8ada7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:12.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.533+0000 7f8ada7fc640 10 monclient(hunting): my global_id is 4396 2024-03-25T18:12:12.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.533+0000 7f8ada7fc640 1 -- 172.21.15.156:0/2990677225 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ac4003520 con 0x7f8adc1430c0 2024-03-25T18:12:12.540 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.539+0000 7f8ada7fc640 1 -- 172.21.15.156:0/2990677225 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250610717 0 0) 0x7f8adc142520 con 0x7f8adc0a98f0 2024-03-25T18:12:12.540 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.539+0000 7f8ada7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:12.540 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.539+0000 7f8ada7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:12.540 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.539+0000 7f8ada7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:12.540 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.539+0000 7f8ada7fc640 10 monclient(hunting): my global_id is 4398 2024-03-25T18:12:12.540 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.539+0000 7f8ada7fc640 1 -- 172.21.15.156:0/2990677225 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8adc0b28a0 con 0x7f8adc0a98f0 2024-03-25T18:12:12.546 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.545+0000 7f8ada7fc640 1 -- 172.21.15.156:0/2990677225 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266636378 0 0) 0x7f8adc0b2a30 con 0x7f8adc13f710 2024-03-25T18:12:12.546 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.545+0000 7f8ada7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:12.546 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.545+0000 7f8ada7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:12.546 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.545+0000 7f8ada7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:12.546 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.545+0000 7f8ada7fc640 10 monclient(hunting): my global_id is 4406 2024-03-25T18:12:12.546 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.545+0000 7f8ada7fc640 1 -- 172.21.15.156:0/2990677225 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8adc142520 con 0x7f8adc13f710 2024-03-25T18:12:12.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.545+0000 7f8ada7fc640 1 -- 172.21.15.156:0/2990677225 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4111217761 0 0) 0x7f8ac4003520 con 0x7f8adc1430c0 2024-03-25T18:12:12.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.546+0000 7f8ada7fc640 1 -- 172.21.15.156:0/2990677225 >> v1:172.21.15.156:6790/0 conn(0x7f8adc13f710 legacy=0x7f8adc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:12.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.546+0000 7f8ada7fc640 1 -- 172.21.15.156:0/2990677225 >> v1:172.21.15.156:6789/0 conn(0x7f8adc0a98f0 legacy=0x7f8adc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:12.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.546+0000 7f8ada7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:12:12.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.546+0000 7f8ada7fc640 1 monclient: found mon.noname-c 2024-03-25T18:12:12.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.546+0000 7f8ada7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:12.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.546+0000 7f8ada7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:12:12.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.546+0000 7f8ada7fc640 1 -- 172.21.15.156:0/2990677225 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8adc0b44c0 con 0x7f8adc1430c0 2024-03-25T18:12:12.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.546+0000 7f8ada7fc640 10 monclient: _finish_auth 0 2024-03-25T18:12:12.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.546+0000 7f8ada7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:12:12.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.546+0000 7f8ada7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:12.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.546+0000 7f8ada7fc640 1 -- 172.21.15.156:0/2990677225 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f8ad4001a10 con 0x7f8adc1430c0 2024-03-25T18:12:12.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.546+0000 7f8ae1e55640 5 monclient: authenticate success, global_id 4396 2024-03-25T18:12:12.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.546+0000 7f8ada7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:12.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.546+0000 7f8ada7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:12.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.546+0000 7f8ada7fc640 10 monclient: dump: 2024-03-25T18:12:12.548 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:12.548 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:12.548 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:12.548 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:12.548 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:12.548 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:12.548 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:12.549 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:12.549 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:12.549 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:12.549 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.546+0000 7f8ae1e55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:12.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.553+0000 7f8ada7fc640 1 -- 172.21.15.156:0/2990677225 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ad4001b70 con 0x7f8adc1430c0 2024-03-25T18:12:12.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.553+0000 7f8ada7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:12.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.553+0000 7f8ada7fc640 1 -- 172.21.15.156:0/2990677225 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f8ad4001f80 con 0x7f8adc1430c0 2024-03-25T18:12:12.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.553+0000 7f8ae1e55640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:12.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.553+0000 7f8ae1e55640 4 set_mon_vals no callback set 2024-03-25T18:12:12.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.553+0000 7f8ada7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:12.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.553+0000 7f8ada7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:12.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.553+0000 7f8ada7fc640 10 monclient: dump: 2024-03-25T18:12:12.554 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:12.555 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:12.555 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:12.555 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:12.555 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:12.555 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:12.555 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:12.555 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:12.555 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:12.555 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:12.556 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.554+0000 7f8ae1e55640 10 monclient: shutdown 2024-03-25T18:12:12.557 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.554+0000 7f8ae1e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:12.557 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.554+0000 7f8ae1e55640 1 -- 172.21.15.156:0/2990677225 >> v1:172.21.15.194:6789/0 conn(0x7f8adc1430c0 legacy=0x7f8adc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:12.557 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.554+0000 7f8ae1e55640 1 -- 172.21.15.156:0/2990677225 shutdown_connections 2024-03-25T18:12:12.557 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.554+0000 7f8ae1e55640 1 -- 172.21.15.156:0/2990677225 >> 172.21.15.156:0/2990677225 conn(0x7f8adc0b7ea0 msgr2=0x7f8adc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:12.557 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.554+0000 7f8ae1e55640 1 -- 172.21.15.156:0/2990677225 shutdown_connections 2024-03-25T18:12:12.557 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.554+0000 7f8ae1e55640 1 -- 172.21.15.156:0/2990677225 wait complete. 2024-03-25T18:12:12.557 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.555+0000 7f8ae1e55640 10 monclient: build_initial_monmap 2024-03-25T18:12:12.557 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.555+0000 7f8ae1e55640 10 monclient: monmap: 2024-03-25T18:12:12.557 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:12:12.557 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:12.557 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:12:12.557 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:12:12.557 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:12.557 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:12:12.557 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:12:12.558 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:12:12.558 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:12:12.558 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:12.558 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.555+0000 7f8ae1e55640 1 librados: starting msgr at 2024-03-25T18:12:12.558 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.555+0000 7f8ae1e55640 1 librados: starting objecter 2024-03-25T18:12:12.558 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.555+0000 7f8ae1e55640 1 Processor -- start 2024-03-25T18:12:12.558 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.555+0000 7f8ae1e55640 1 -- start start 2024-03-25T18:12:12.558 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.555+0000 7f8ae1e55640 1 librados: setting wanted keys 2024-03-25T18:12:12.558 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.555+0000 7f8ae1e55640 1 librados: calling monclient init 2024-03-25T18:12:12.558 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.555+0000 7f8ae1e55640 10 monclient: init 2024-03-25T18:12:12.559 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.556+0000 7f8ae1e55640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:12.559 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.556+0000 7f8ae1e55640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:12:12.559 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.556+0000 7f8ae1e55640 10 monclient(hunting): picked mon.noname-b con 0x7f8adc0a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:12:12.559 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.556+0000 7f8ae1e55640 10 monclient(hunting): picked mon.noname-c con 0x7f8adc13f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:12:12.559 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.556+0000 7f8ae1e55640 10 monclient(hunting): picked mon.noname-a con 0x7f8adc1430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:12:12.559 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.556+0000 7f8ae1e55640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8adc1d7500 con 0x7f8adc1430c0 2024-03-25T18:12:12.559 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.556+0000 7f8ae1e55640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8adc1394a0 con 0x7f8adc0a98f0 2024-03-25T18:12:12.560 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.556+0000 7f8ae1e55640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8adc138b50 con 0x7f8adc13f710 2024-03-25T18:12:12.560 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.556+0000 7f8ae1e55640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:12.560 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.556+0000 7f8ae1e55640 10 monclient(hunting): authenticate will time out at 7213.867558s 2024-03-25T18:12:12.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.565+0000 7f8adbfff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f8adc1430c0 0x7f8adc134110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:48838/0 (socket says 172.21.15.156:48838) 2024-03-25T18:12:12.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.565+0000 7f8adbfff640 1 -- 172.21.15.156:0/1430787266 learned_addr learned my addr 172.21.15.156:0/1430787266 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:12:12.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.567+0000 7f8ac3fff640 10 client.?.objecter ms_handle_connect 0x7f8adc1430c0 2024-03-25T18:12:12.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.567+0000 7f8ac3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:12.573 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.573+0000 7f8ac3fff640 10 client.?.objecter ms_handle_connect 0x7f8adc0a98f0 2024-03-25T18:12:12.573 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.573+0000 7f8ac3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:12.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.578+0000 7f8ac3fff640 1 -- 172.21.15.156:0/1430787266 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989419197 0 0) 0x7f8adc1d7500 con 0x7f8adc1430c0 2024-03-25T18:12:12.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.578+0000 7f8ac3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:12.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.578+0000 7f8ac3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:12.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.578+0000 7f8ac3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:12.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.578+0000 7f8ac3fff640 10 monclient(hunting): my global_id is 4401 2024-03-25T18:12:12.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.579+0000 7f8ac3fff640 1 -- 172.21.15.156:0/1430787266 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ac8004eb0 con 0x7f8adc1430c0 2024-03-25T18:12:12.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.579+0000 7f8ac3fff640 10 client.?.objecter ms_handle_connect 0x7f8adc13f710 2024-03-25T18:12:12.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.579+0000 7f8ac3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:12.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.589+0000 7f8ac3fff640 1 -- 172.21.15.156:0/1430787266 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745304216 0 0) 0x7f8adc138b50 con 0x7f8adc13f710 2024-03-25T18:12:12.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.589+0000 7f8ac3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:12.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.589+0000 7f8ac3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:12.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.589+0000 7f8ac3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:12.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.589+0000 7f8ac3fff640 10 monclient(hunting): my global_id is 4409 2024-03-25T18:12:12.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.589+0000 7f8ac3fff640 1 -- 172.21.15.156:0/1430787266 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8adc1d7500 con 0x7f8adc13f710 2024-03-25T18:12:12.595 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.594+0000 7f8ac3fff640 1 -- 172.21.15.156:0/1430787266 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3295027270 0 0) 0x7f8adc1394a0 con 0x7f8adc0a98f0 2024-03-25T18:12:12.595 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.594+0000 7f8ac3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:12.595 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.594+0000 7f8ac3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:12.595 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.594+0000 7f8ac3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:12.595 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.594+0000 7f8ac3fff640 10 monclient(hunting): my global_id is 4399 2024-03-25T18:12:12.595 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.594+0000 7f8ac3fff640 1 -- 172.21.15.156:0/1430787266 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8adc138b50 con 0x7f8adc0a98f0 2024-03-25T18:12:12.598 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.597+0000 7f8ac3fff640 1 -- 172.21.15.156:0/1430787266 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1403623484 0 0) 0x7f8ac8004eb0 con 0x7f8adc1430c0 2024-03-25T18:12:12.598 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.597+0000 7f8ac3fff640 1 -- 172.21.15.156:0/1430787266 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8adc1394a0 con 0x7f8adc1430c0 2024-03-25T18:12:12.598 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.597+0000 7f8ac3fff640 1 -- 172.21.15.156:0/1430787266 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f8ad4001ec0 con 0x7f8adc1430c0 2024-03-25T18:12:12.598 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.597+0000 7f8ac3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:12.598 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.597+0000 7f8ac3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:12.598 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.597+0000 7f8ac3fff640 10 monclient(hunting): dump: 2024-03-25T18:12:12.599 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:12.599 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:12.599 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:12.599 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:12.599 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:12.599 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:12.599 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:12.599 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:12.599 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:12.599 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:12.599 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.597+0000 7f8ac3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.605+0000 7f8ac3fff640 1 -- 172.21.15.156:0/1430787266 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1210290492 0 0) 0x7f8adc1394a0 con 0x7f8adc1430c0 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.605+0000 7f8ac3fff640 1 -- 172.21.15.156:0/1430787266 >> v1:172.21.15.156:6790/0 conn(0x7f8adc13f710 legacy=0x7f8adc1d7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.605+0000 7f8ac3fff640 1 -- 172.21.15.156:0/1430787266 >> v1:172.21.15.194:6789/0 conn(0x7f8adc0a98f0 legacy=0x7f8adc138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.605+0000 7f8ac3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.605+0000 7f8ac3fff640 1 monclient: found mon.a 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.605+0000 7f8ac3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.605+0000 7f8ac3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.605+0000 7f8ac3fff640 1 -- 172.21.15.156:0/1430787266 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8adc1d8380 con 0x7f8adc1430c0 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.605+0000 7f8ac3fff640 10 monclient: _finish_auth 0 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.605+0000 7f8ac3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.605+0000 7f8ac3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.605+0000 7f8ae1e55640 5 monclient: authenticate success, global_id 4401 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.605+0000 7f8ae1e55640 10 monclient: _renew_subs 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.605+0000 7f8ae1e55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:12:12.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.605+0000 7f8ae1e55640 1 -- 172.21.15.156:0/1430787266 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8adc139710 con 0x7f8adc1430c0 2024-03-25T18:12:12.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.605+0000 7f8ae1e55640 10 client.4401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:12.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.605+0000 7f8ae1e55640 10 monclient: _renew_subs 2024-03-25T18:12:12.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.605+0000 7f8ae1e55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:12:12.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.605+0000 7f8ae1e55640 1 -- 172.21.15.156:0/1430787266 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8adc1d7750 con 0x7f8adc1430c0 2024-03-25T18:12:12.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.605+0000 7f8ae1e55640 1 librados: init done 2024-03-25T18:12:12.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.606+0000 7f8ae1e55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:12.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.606+0000 7f8ae1e55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:12.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.606+0000 7f8ae1e55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:12:12.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.606+0000 7f8ae1e55640 1 -- 172.21.15.156:0/1430787266 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8adc0627b0 con 0x7f8adc1430c0 2024-03-25T18:12:12.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.904+0000 7f8ac3fff640 1 -- 172.21.15.156:0/1430787266 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ad4003c20 con 0x7f8adc1430c0 2024-03-25T18:12:12.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.904+0000 7f8ac3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:12.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.904+0000 7f8ac3fff640 1 -- 172.21.15.156:0/1430787266 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f8ad4002100 con 0x7f8adc1430c0 2024-03-25T18:12:12.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.904+0000 7f8ad9ffb640 4 set_mon_vals no callback set 2024-03-25T18:12:12.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.904+0000 7f8ac3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:12.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.904+0000 7f8ac3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:12.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.904+0000 7f8ac3fff640 10 monclient: dump: 2024-03-25T18:12:12.906 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:12.906 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:12.906 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:12.907 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:12.907 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:12.907 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:12.907 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:12.907 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:12.907 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:12.907 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:12.907 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.905+0000 7f8ac3fff640 1 -- 172.21.15.156:0/1430787266 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f8ad40040f0 con 0x7f8adc1430c0 2024-03-25T18:12:12.908 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.906+0000 7f8adb7fe640 10 monclient: get_auth_request con 0x7f8ac806de00 auth_method 0 2024-03-25T18:12:12.908 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.906+0000 7f8ac3fff640 10 client.4401.objecter ms_handle_connect 0x7f8ac806de00 2024-03-25T18:12:12.917 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.916+0000 7f8ac3fff640 1 -- 172.21.15.156:0/1430787266 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f8ad4054af0 con 0x7f8adc1430c0 2024-03-25T18:12:12.917 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.916+0000 7f8ac3fff640 10 client.4401.objecter ms_dispatch 0x7f8adc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:12.917 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.916+0000 7f8ac3fff640 3 client.4401.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:12.917 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.916+0000 7f8ac3fff640 3 client.4401.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:12.917 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.916+0000 7f8ac3fff640 20 client.4401.objecter dump_active .. 0 homeless 2024-03-25T18:12:12.917 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.916+0000 7f8ac3fff640 1 -- 172.21.15.156:0/1430787266 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8ad4050600 con 0x7f8adc1430c0 2024-03-25T18:12:12.917 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.916+0000 7f8ac3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:12.917 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:12.916+0000 7f8ac3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:13.174 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:13.173+0000 7f8ae1e55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:13.174 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:13.173+0000 7f8ae1e55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:13.174 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:13.173+0000 7f8ae1e55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:12:13.174 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:13.173+0000 7f8ae1e55640 1 -- 172.21.15.156:0/1430787266 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8adc13fdc0 con 0x7f8adc1430c0 2024-03-25T18:12:13.201 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:13.200+0000 7f8ac3fff640 1 -- 172.21.15.156:0/1430787266 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f8ad4052530 con 0x7f8adc1430c0 2024-03-25T18:12:13.201 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:13.200+0000 7f8ac3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:13.201 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:13.200+0000 7f8ac3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:13.202 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:12:13.202 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:13.205 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:13.203+0000 7f8ae1e55640 10 librados: watch_flush enter 2024-03-25T18:12:13.205 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:13.203+0000 7f8ae1e55640 10 librados: watch_flush exit 2024-03-25T18:12:13.205 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:13.203+0000 7f8ae1e55640 20 client.4401.objecter shutdown clearing up homeless session... 2024-03-25T18:12:13.205 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:13.203+0000 7f8ae1e55640 10 client.4401.objecter successfully canceled tick 2024-03-25T18:12:13.205 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:13.203+0000 7f8ae1e55640 1 -- 172.21.15.156:0/1430787266 >> v1:172.21.15.194:6800/2946427713 conn(0x7f8ac806de00 legacy=0x7f8ac80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:13.205 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:13.203+0000 7f8ae1e55640 10 monclient: shutdown 2024-03-25T18:12:13.205 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:13.203+0000 7f8ae1e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:13.206 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:13.203+0000 7f8ae1e55640 1 -- 172.21.15.156:0/1430787266 >> v1:172.21.15.156:6789/0 conn(0x7f8adc1430c0 legacy=0x7f8adc134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:13.206 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:13.203+0000 7f8ae1e55640 1 -- 172.21.15.156:0/1430787266 shutdown_connections 2024-03-25T18:12:13.206 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:13.203+0000 7f8ae1e55640 1 -- 172.21.15.156:0/1430787266 >> 172.21.15.156:0/1430787266 conn(0x7f8adc0b7ea0 msgr2=0x7f8adc13d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:13.206 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:13.203+0000 7f8ae1e55640 1 -- 172.21.15.156:0/1430787266 shutdown_connections 2024-03-25T18:12:13.206 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:13.203+0000 7f8ae1e55640 1 -- 172.21.15.156:0/1430787266 wait complete. 2024-03-25T18:12:13.206 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:13.203+0000 7f8ae1e55640 1 librados: shutdown 2024-03-25T18:12:13.217 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:20.024 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:20.139 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.138+0000 7f925d238640 10 monclient: get_monmap_and_config 2024-03-25T18:12:20.139 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.138+0000 7f925d238640 10 monclient: build_initial_monmap 2024-03-25T18:12:20.139 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.138+0000 7f925d238640 10 monclient: monmap: 2024-03-25T18:12:20.139 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:12:20.139 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:20.139 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:12:20.139532+0000 2024-03-25T18:12:20.139 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:12:20.139532+0000 2024-03-25T18:12:20.139 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:20.140 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:20.140 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:12:20.140 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:12:20.140 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:12:20.140 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:20.141 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.139+0000 7f925d238640 1 Processor -- start 2024-03-25T18:12:20.141 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.139+0000 7f925d238640 1 -- start start 2024-03-25T18:12:20.141 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.139+0000 7f925d238640 10 monclient: init 2024-03-25T18:12:20.141 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.140+0000 7f925d238640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:20.141 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.140+0000 7f925d238640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:12:20.141 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.140+0000 7f925d238640 10 monclient(hunting): picked mon.noname-b con 0x7f92580b4d60 addr v1:172.21.15.156:6790/0 2024-03-25T18:12:20.141 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.140+0000 7f925d238640 10 monclient(hunting): picked mon.noname-c con 0x7f92580b1730 addr v1:172.21.15.194:6789/0 2024-03-25T18:12:20.141 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.140+0000 7f925d238640 10 monclient(hunting): picked mon.noname-a con 0x7f925814be90 addr v1:172.21.15.156:6789/0 2024-03-25T18:12:20.141 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.140+0000 7f925d238640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92580af940 con 0x7f925814be90 2024-03-25T18:12:20.141 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.140+0000 7f925d238640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92580b0a70 con 0x7f92580b1730 2024-03-25T18:12:20.141 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.140+0000 7f925d238640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92580b0d50 con 0x7f92580b4d60 2024-03-25T18:12:20.141 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.140+0000 7f925d238640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:20.141 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.140+0000 7f925d238640 10 monclient(hunting): authenticate will time out at 7221.451089s 2024-03-25T18:12:20.146 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.145+0000 7f9256d9d640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f92580b4d60 0x7f925813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:49136/0 (socket says 172.21.15.156:49136) 2024-03-25T18:12:20.146 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.145+0000 7f9256d9d640 1 -- 172.21.15.156:0/1067763795 learned_addr learned my addr 172.21.15.156:0/1067763795 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:12:20.157 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.156+0000 7f925659c640 1 -- 172.21.15.156:0/1067763795 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2273904869 0 0) 0x7f92580b0d50 con 0x7f92580b4d60 2024-03-25T18:12:20.157 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.156+0000 7f925659c640 10 monclient(hunting): _negotiate 2024-03-25T18:12:20.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.156+0000 7f925659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:20.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.156+0000 7f925659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:20.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.156+0000 7f925659c640 10 monclient(hunting): my global_id is 4412 2024-03-25T18:12:20.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.156+0000 7f925659c640 1 -- 172.21.15.156:0/1067763795 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9244003520 con 0x7f92580b4d60 2024-03-25T18:12:20.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.167+0000 7f925659c640 1 -- 172.21.15.156:0/1067763795 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1943048911 0 0) 0x7f92580b0a70 con 0x7f92580b1730 2024-03-25T18:12:20.168 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.167+0000 7f925659c640 10 monclient(hunting): _negotiate 2024-03-25T18:12:20.168 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.167+0000 7f925659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:20.168 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.167+0000 7f925659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:20.168 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.167+0000 7f925659c640 10 monclient(hunting): my global_id is 4402 2024-03-25T18:12:20.168 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.167+0000 7f925659c640 1 -- 172.21.15.156:0/1067763795 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92580b0d50 con 0x7f92580b1730 2024-03-25T18:12:20.168 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.167+0000 7f925659c640 1 -- 172.21.15.156:0/1067763795 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1950936756 0 0) 0x7f92580af940 con 0x7f925814be90 2024-03-25T18:12:20.168 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.167+0000 7f925659c640 10 monclient(hunting): _negotiate 2024-03-25T18:12:20.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.167+0000 7f925659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:20.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.167+0000 7f925659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:20.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.167+0000 7f925659c640 10 monclient(hunting): my global_id is 4404 2024-03-25T18:12:20.169 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.167+0000 7f925659c640 1 -- 172.21.15.156:0/1067763795 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92580b0a70 con 0x7f925814be90 2024-03-25T18:12:20.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.176+0000 7f925659c640 1 -- 172.21.15.156:0/1067763795 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 204725823 0 0) 0x7f92580b0d50 con 0x7f92580b1730 2024-03-25T18:12:20.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.176+0000 7f925659c640 1 -- 172.21.15.156:0/1067763795 >> v1:172.21.15.156:6790/0 conn(0x7f92580b4d60 legacy=0x7f925813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:20.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.176+0000 7f925659c640 1 -- 172.21.15.156:0/1067763795 >> v1:172.21.15.156:6789/0 conn(0x7f925814be90 legacy=0x7f925814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:20.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.176+0000 7f925659c640 10 monclient: _finish_hunting 0 2024-03-25T18:12:20.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.176+0000 7f925659c640 1 monclient: found mon.noname-c 2024-03-25T18:12:20.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.176+0000 7f925659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:20.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.176+0000 7f925659c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:12:20.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.176+0000 7f925659c640 1 -- 172.21.15.156:0/1067763795 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92580b1030 con 0x7f92580b1730 2024-03-25T18:12:20.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.176+0000 7f925659c640 10 monclient: _finish_auth 0 2024-03-25T18:12:20.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.176+0000 7f925659c640 10 monclient: _check_auth_tickets 2024-03-25T18:12:20.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.176+0000 7f925659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:20.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.176+0000 7f925d238640 5 monclient: authenticate success, global_id 4402 2024-03-25T18:12:20.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.176+0000 7f925d238640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:20.183 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.183+0000 7f925659c640 1 -- 172.21.15.156:0/1067763795 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9240001630 con 0x7f92580b1730 2024-03-25T18:12:20.184 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.183+0000 7f925659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:20.184 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.183+0000 7f925659c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:20.184 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.183+0000 7f925659c640 10 monclient: dump: 2024-03-25T18:12:20.184 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:20.184 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:20.184 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:20.184 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:20.184 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:20.184 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:20.184 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:20.184 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:20.184 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:20.184 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:20.184 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.183+0000 7f925d238640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:20.202 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.201+0000 7f925659c640 1 -- 172.21.15.156:0/1067763795 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9240001880 con 0x7f92580b1730 2024-03-25T18:12:20.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.201+0000 7f925659c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:20.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.201+0000 7f925659c640 1 -- 172.21.15.156:0/1067763795 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9240001c10 con 0x7f92580b1730 2024-03-25T18:12:20.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.201+0000 7f925d238640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:20.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.202+0000 7f925d238640 4 set_mon_vals no callback set 2024-03-25T18:12:20.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.202+0000 7f925659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:20.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.202+0000 7f925659c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:20.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.202+0000 7f925659c640 10 monclient: dump: 2024-03-25T18:12:20.203 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:20.203 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:20.203 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:20.203 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:20.203 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:20.203 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:20.203 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:20.203 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:20.204 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:20.204 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:20.206 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.202+0000 7f925d238640 10 monclient: shutdown 2024-03-25T18:12:20.206 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.202+0000 7f925d238640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:20.206 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.202+0000 7f925d238640 1 -- 172.21.15.156:0/1067763795 >> v1:172.21.15.194:6789/0 conn(0x7f92580b1730 legacy=0x7f92580b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:20.206 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.202+0000 7f925d238640 1 -- 172.21.15.156:0/1067763795 shutdown_connections 2024-03-25T18:12:20.206 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.202+0000 7f925d238640 1 -- 172.21.15.156:0/1067763795 >> 172.21.15.156:0/1067763795 conn(0x7f92580617a0 msgr2=0x7f9258061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:20.206 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.202+0000 7f925d238640 1 -- 172.21.15.156:0/1067763795 shutdown_connections 2024-03-25T18:12:20.206 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.202+0000 7f925d238640 1 -- 172.21.15.156:0/1067763795 wait complete. 2024-03-25T18:12:20.206 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.203+0000 7f925d238640 10 monclient: build_initial_monmap 2024-03-25T18:12:20.206 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.203+0000 7f925d238640 10 monclient: monmap: 2024-03-25T18:12:20.206 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:12:20.206 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:20.206 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:12:20.206 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:12:20.206 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:20.206 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:12:20.207 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:12:20.207 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:12:20.207 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:12:20.207 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:20.207 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.204+0000 7f925d238640 1 librados: starting msgr at 2024-03-25T18:12:20.207 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.204+0000 7f925d238640 1 librados: starting objecter 2024-03-25T18:12:20.207 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.204+0000 7f925d238640 1 Processor -- start 2024-03-25T18:12:20.207 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.204+0000 7f925d238640 1 -- start start 2024-03-25T18:12:20.207 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.204+0000 7f925d238640 1 librados: setting wanted keys 2024-03-25T18:12:20.207 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.204+0000 7f925d238640 1 librados: calling monclient init 2024-03-25T18:12:20.207 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.204+0000 7f925d238640 10 monclient: init 2024-03-25T18:12:20.207 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.204+0000 7f925d238640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:20.207 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.204+0000 7f925d238640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:12:20.207 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.204+0000 7f925d238640 10 monclient(hunting): picked mon.noname-c con 0x7f92580b1730 addr v1:172.21.15.156:6790/0 2024-03-25T18:12:20.207 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.204+0000 7f925d238640 10 monclient(hunting): picked mon.noname-b con 0x7f92580b4d60 addr v1:172.21.15.194:6789/0 2024-03-25T18:12:20.207 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.205+0000 7f925d238640 10 monclient(hunting): picked mon.noname-a con 0x7f925814be90 addr v1:172.21.15.156:6789/0 2024-03-25T18:12:20.208 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.205+0000 7f925d238640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92581d7910 con 0x7f925814be90 2024-03-25T18:12:20.208 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.205+0000 7f925d238640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92581d7bf0 con 0x7f92580b4d60 2024-03-25T18:12:20.208 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.205+0000 7f925d238640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9258136cf0 con 0x7f92580b1730 2024-03-25T18:12:20.208 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.205+0000 7f925d238640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:20.208 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.205+0000 7f925d238640 10 monclient(hunting): authenticate will time out at 7221.515902s 2024-03-25T18:12:20.212 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.211+0000 7f925759e640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f925814be90 0x7f92581e2370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:53652/0 (socket says 172.21.15.156:53652) 2024-03-25T18:12:20.212 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.211+0000 7f925759e640 1 -- 172.21.15.156:0/1703197602 learned_addr learned my addr 172.21.15.156:0/1703197602 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:12:20.216 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.215+0000 7f924f7fe640 10 client.?.objecter ms_handle_connect 0x7f92580b4d60 2024-03-25T18:12:20.216 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.215+0000 7f924f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:20.216 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.215+0000 7f924f7fe640 10 client.?.objecter ms_handle_connect 0x7f925814be90 2024-03-25T18:12:20.216 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.215+0000 7f924f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:20.218 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.217+0000 7f924f7fe640 10 client.?.objecter ms_handle_connect 0x7f92580b1730 2024-03-25T18:12:20.218 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.217+0000 7f924f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:20.222 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.221+0000 7f924f7fe640 1 -- 172.21.15.156:0/1703197602 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 648071672 0 0) 0x7f92581d7bf0 con 0x7f92580b4d60 2024-03-25T18:12:20.222 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.221+0000 7f924f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:20.223 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.221+0000 7f924f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:20.223 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.221+0000 7f924f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:20.223 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.221+0000 7f924f7fe640 10 monclient(hunting): my global_id is 4405 2024-03-25T18:12:20.223 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.221+0000 7f924f7fe640 1 -- 172.21.15.156:0/1703197602 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9238004eb0 con 0x7f92580b4d60 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.232+0000 7f924f7fe640 1 -- 172.21.15.156:0/1703197602 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3366764370 0 0) 0x7f9238004eb0 con 0x7f92580b4d60 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.232+0000 7f924f7fe640 1 -- 172.21.15.156:0/1703197602 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92581d7bf0 con 0x7f92580b4d60 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.232+0000 7f924f7fe640 1 -- 172.21.15.156:0/1703197602 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9240001b90 con 0x7f92580b4d60 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.232+0000 7f924f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.232+0000 7f924f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.232+0000 7f924f7fe640 10 monclient(hunting): dump: 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:20.234 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.232+0000 7f924f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:20.234 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.232+0000 7f924f7fe640 1 -- 172.21.15.156:0/1703197602 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736573277 0 0) 0x7f9258136cf0 con 0x7f92580b1730 2024-03-25T18:12:20.234 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.233+0000 7f924f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:20.234 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.233+0000 7f924f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:20.234 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.233+0000 7f924f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:20.234 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.233+0000 7f924f7fe640 10 monclient(hunting): my global_id is 4415 2024-03-25T18:12:20.234 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.233+0000 7f924f7fe640 1 -- 172.21.15.156:0/1703197602 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9238004eb0 con 0x7f92580b1730 2024-03-25T18:12:20.237 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.236+0000 7f924f7fe640 1 -- 172.21.15.156:0/1703197602 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804252900 0 0) 0x7f92581d7910 con 0x7f925814be90 2024-03-25T18:12:20.237 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.236+0000 7f924f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:20.237 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.236+0000 7f924f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:20.237 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.236+0000 7f924f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:20.237 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.236+0000 7f924f7fe640 10 monclient(hunting): my global_id is 4407 2024-03-25T18:12:20.237 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.236+0000 7f924f7fe640 1 -- 172.21.15.156:0/1703197602 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9258136cf0 con 0x7f925814be90 2024-03-25T18:12:20.244 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.243+0000 7f924f7fe640 1 -- 172.21.15.156:0/1703197602 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1682575527 0 0) 0x7f92581d7bf0 con 0x7f92580b4d60 2024-03-25T18:12:20.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.243+0000 7f924f7fe640 1 -- 172.21.15.156:0/1703197602 >> v1:172.21.15.156:6790/0 conn(0x7f92580b1730 legacy=0x7f92581d8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:20.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.243+0000 7f924f7fe640 1 -- 172.21.15.156:0/1703197602 >> v1:172.21.15.156:6789/0 conn(0x7f925814be90 legacy=0x7f92581e2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:20.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.243+0000 7f924f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:12:20.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.243+0000 7f924f7fe640 1 monclient: found mon.b 2024-03-25T18:12:20.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.243+0000 7f924f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:20.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.243+0000 7f924f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:12:20.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.243+0000 7f924f7fe640 1 -- 172.21.15.156:0/1703197602 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9258136fd0 con 0x7f92580b4d60 2024-03-25T18:12:20.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.243+0000 7f924f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:12:20.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.243+0000 7f924f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:12:20.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.243+0000 7f924f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:20.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.243+0000 7f925d238640 5 monclient: authenticate success, global_id 4405 2024-03-25T18:12:20.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.243+0000 7f925d238640 10 monclient: _renew_subs 2024-03-25T18:12:20.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.243+0000 7f925d238640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:12:20.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.243+0000 7f925d238640 1 -- 172.21.15.156:0/1703197602 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9258137220 con 0x7f92580b4d60 2024-03-25T18:12:20.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.243+0000 7f925d238640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:20.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.243+0000 7f925d238640 10 monclient: _renew_subs 2024-03-25T18:12:20.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.243+0000 7f925d238640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:12:20.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.243+0000 7f925d238640 1 -- 172.21.15.156:0/1703197602 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9258143ae0 con 0x7f92580b4d60 2024-03-25T18:12:20.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.243+0000 7f925d238640 1 librados: init done 2024-03-25T18:12:20.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.245+0000 7f925d238640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:20.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.245+0000 7f925d238640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:20.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.245+0000 7f925d238640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:12:20.247 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.245+0000 7f925d238640 1 -- 172.21.15.156:0/1703197602 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92580b5e70 con 0x7f92580b4d60 2024-03-25T18:12:20.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.534+0000 7f924f7fe640 1 -- 172.21.15.156:0/1703197602 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92400038a0 con 0x7f92580b4d60 2024-03-25T18:12:20.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.534+0000 7f924f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:20.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.534+0000 7f924f7fe640 1 -- 172.21.15.156:0/1703197602 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9240003cb0 con 0x7f92580b4d60 2024-03-25T18:12:20.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.534+0000 7f924f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:20.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.534+0000 7f924f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:20.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.534+0000 7f924f7fe640 10 monclient: dump: 2024-03-25T18:12:20.535 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:20.535 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:20.535 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:20.535 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:20.535 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:20.535 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:20.535 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:20.536 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:20.536 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:20.536 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:20.536 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.534+0000 7f9255d9b640 4 set_mon_vals no callback set 2024-03-25T18:12:20.536 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.534+0000 7f924f7fe640 1 -- 172.21.15.156:0/1703197602 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f9240003ed0 con 0x7f92580b4d60 2024-03-25T18:12:20.536 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.535+0000 7f9256d9d640 10 monclient: get_auth_request con 0x7f923806de10 auth_method 0 2024-03-25T18:12:20.536 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.535+0000 7f924f7fe640 10 client.4405.objecter ms_handle_connect 0x7f923806de10 2024-03-25T18:12:20.541 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.540+0000 7f924f7fe640 1 -- 172.21.15.156:0/1703197602 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f92400536e0 con 0x7f92580b4d60 2024-03-25T18:12:20.541 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.540+0000 7f924f7fe640 10 client.4405.objecter ms_dispatch 0x7f9258000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:20.541 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.540+0000 7f924f7fe640 3 client.4405.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:20.541 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.540+0000 7f924f7fe640 3 client.4405.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:20.541 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.540+0000 7f924f7fe640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-25T18:12:20.542 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.541+0000 7f924f7fe640 1 -- 172.21.15.156:0/1703197602 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f924004f1f0 con 0x7f92580b4d60 2024-03-25T18:12:20.542 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.541+0000 7f924f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:20.542 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.541+0000 7f924f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:20.749 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.748+0000 7f925d238640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:20.750 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.748+0000 7f925d238640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:20.750 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.748+0000 7f925d238640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:12:20.750 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.748+0000 7f925d238640 1 -- 172.21.15.156:0/1703197602 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92580627b0 con 0x7f92580b4d60 2024-03-25T18:12:20.841 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.840+0000 7f924f7fe640 1 -- 172.21.15.156:0/1703197602 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f9240051120 con 0x7f92580b4d60 2024-03-25T18:12:20.841 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.840+0000 7f924f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:20.841 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.840+0000 7f924f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:20.841 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:12:20.841 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:20.844 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.842+0000 7f925d238640 10 librados: watch_flush enter 2024-03-25T18:12:20.844 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.842+0000 7f925d238640 10 librados: watch_flush exit 2024-03-25T18:12:20.844 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.842+0000 7f925d238640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-25T18:12:20.844 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.842+0000 7f925d238640 10 client.4405.objecter successfully canceled tick 2024-03-25T18:12:20.844 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.843+0000 7f925d238640 1 -- 172.21.15.156:0/1703197602 >> v1:172.21.15.194:6800/2946427713 conn(0x7f923806de10 legacy=0x7f92380702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:20.845 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.843+0000 7f925d238640 10 monclient: shutdown 2024-03-25T18:12:20.845 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.843+0000 7f925d238640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:20.845 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.843+0000 7f925d238640 1 -- 172.21.15.156:0/1703197602 >> v1:172.21.15.194:6789/0 conn(0x7f92580b4d60 legacy=0x7f92581365e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:20.845 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.843+0000 7f925d238640 1 -- 172.21.15.156:0/1703197602 shutdown_connections 2024-03-25T18:12:20.845 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.843+0000 7f925d238640 1 -- 172.21.15.156:0/1703197602 >> 172.21.15.156:0/1703197602 conn(0x7f92580617a0 msgr2=0x7f92580b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:20.845 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.843+0000 7f925d238640 1 -- 172.21.15.156:0/1703197602 shutdown_connections 2024-03-25T18:12:20.845 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.843+0000 7f925d238640 1 -- 172.21.15.156:0/1703197602 wait complete. 2024-03-25T18:12:20.845 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:20.843+0000 7f925d238640 1 librados: shutdown 2024-03-25T18:12:20.855 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:27.661 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:27.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.776+0000 7fe9200c8640 10 monclient: get_monmap_and_config 2024-03-25T18:12:27.778 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.776+0000 7fe9200c8640 10 monclient: build_initial_monmap 2024-03-25T18:12:27.778 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.776+0000 7fe9200c8640 10 monclient: monmap: 2024-03-25T18:12:27.778 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:12:27.778 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:27.778 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:12:27.777779+0000 2024-03-25T18:12:27.778 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:12:27.777779+0000 2024-03-25T18:12:27.778 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:27.779 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:27.779 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:12:27.779 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:12:27.779 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:12:27.779 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:27.780 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.777+0000 7fe9200c8640 1 Processor -- start 2024-03-25T18:12:27.780 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.777+0000 7fe9200c8640 1 -- start start 2024-03-25T18:12:27.780 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.777+0000 7fe9200c8640 10 monclient: init 2024-03-25T18:12:27.780 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.778+0000 7fe9200c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:27.780 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.778+0000 7fe9200c8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:12:27.780 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.778+0000 7fe9200c8640 10 monclient(hunting): picked mon.noname-a con 0x7fe9180a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:12:27.780 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.778+0000 7fe9200c8640 10 monclient(hunting): picked mon.noname-c con 0x7fe91813f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:12:27.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.778+0000 7fe9200c8640 10 monclient(hunting): picked mon.noname-b con 0x7fe9181430c0 addr v1:172.21.15.156:6790/0 2024-03-25T18:12:27.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.778+0000 7fe9200c8640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe918142520 con 0x7fe9180a98f0 2024-03-25T18:12:27.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.778+0000 7fe9200c8640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9180b28a0 con 0x7fe91813f710 2024-03-25T18:12:27.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.778+0000 7fe9200c8640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9180b2a30 con 0x7fe9181430c0 2024-03-25T18:12:27.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.778+0000 7fe9200c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:27.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.778+0000 7fe9200c8640 10 monclient(hunting): authenticate will time out at 7229.089307s 2024-03-25T18:12:27.789 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.789+0000 7fe91d663640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fe91813f710 0x7fe9181429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:41202/0 (socket says 172.21.15.156:41202) 2024-03-25T18:12:27.790 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.789+0000 7fe91d663640 1 -- 172.21.15.156:0/178517110 learned_addr learned my addr 172.21.15.156:0/178517110 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:12:27.797 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.796+0000 7fe91ce62640 1 -- 172.21.15.156:0/178517110 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1512561114 0 0) 0x7fe9180b28a0 con 0x7fe91813f710 2024-03-25T18:12:27.797 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.796+0000 7fe91ce62640 10 monclient(hunting): _negotiate 2024-03-25T18:12:27.797 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.796+0000 7fe91ce62640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:27.797 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.796+0000 7fe91ce62640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:27.797 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.796+0000 7fe91ce62640 10 monclient(hunting): my global_id is 4408 2024-03-25T18:12:27.797 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.796+0000 7fe91ce62640 1 -- 172.21.15.156:0/178517110 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe904003520 con 0x7fe91813f710 2024-03-25T18:12:27.804 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.802+0000 7fe91ce62640 1 -- 172.21.15.156:0/178517110 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2974202131 0 0) 0x7fe904003520 con 0x7fe91813f710 2024-03-25T18:12:27.804 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.803+0000 7fe91ce62640 1 -- 172.21.15.156:0/178517110 >> v1:172.21.15.156:6790/0 conn(0x7fe9181430c0 legacy=0x7fe91814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:12:27.804 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.803+0000 7fe91ce62640 1 -- 172.21.15.156:0/178517110 >> v1:172.21.15.156:6789/0 conn(0x7fe9180a98f0 legacy=0x7fe9180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:27.804 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.803+0000 7fe91ce62640 10 monclient: _finish_hunting 0 2024-03-25T18:12:27.804 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.803+0000 7fe91ce62640 1 monclient: found mon.noname-c 2024-03-25T18:12:27.804 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.803+0000 7fe91ce62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:27.804 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.803+0000 7fe91ce62640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:12:27.804 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.803+0000 7fe91ce62640 1 -- 172.21.15.156:0/178517110 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9180b44c0 con 0x7fe91813f710 2024-03-25T18:12:27.804 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.803+0000 7fe91ce62640 10 monclient: _finish_auth 0 2024-03-25T18:12:27.804 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.803+0000 7fe91ce62640 10 monclient: _check_auth_tickets 2024-03-25T18:12:27.804 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.803+0000 7fe91ce62640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:27.805 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.803+0000 7fe91ce62640 1 -- 172.21.15.156:0/178517110 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fe908001690 con 0x7fe91813f710 2024-03-25T18:12:27.805 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.803+0000 7fe91ce62640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:27.805 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.803+0000 7fe91ce62640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:27.805 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.803+0000 7fe91ce62640 10 monclient: dump: 2024-03-25T18:12:27.805 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:27.805 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:27.805 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:27.805 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:27.805 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:27.805 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:27.805 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:27.805 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:27.805 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:27.805 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:27.805 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.803+0000 7fe9200c8640 5 monclient: authenticate success, global_id 4408 2024-03-25T18:12:27.806 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.803+0000 7fe9200c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:27.815 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.814+0000 7fe91ce62640 1 -- 172.21.15.156:0/178517110 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe908001860 con 0x7fe91813f710 2024-03-25T18:12:27.815 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.814+0000 7fe91ce62640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:27.815 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.814+0000 7fe91ce62640 1 -- 172.21.15.156:0/178517110 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fe908001c70 con 0x7fe91813f710 2024-03-25T18:12:27.815 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.814+0000 7fe91ce62640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:27.816 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.814+0000 7fe91ce62640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:27.816 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.814+0000 7fe91ce62640 10 monclient: dump: 2024-03-25T18:12:27.816 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:27.816 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:27.816 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:27.816 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:27.816 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:27.816 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:27.816 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:27.816 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:27.816 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:27.816 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:27.816 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.814+0000 7fe9200c8640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:27.816 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.814+0000 7fe9200c8640 4 set_mon_vals no callback set 2024-03-25T18:12:27.818 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.815+0000 7fe9200c8640 10 monclient: shutdown 2024-03-25T18:12:27.818 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.815+0000 7fe9200c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:27.819 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.815+0000 7fe9200c8640 1 -- 172.21.15.156:0/178517110 >> v1:172.21.15.194:6789/0 conn(0x7fe91813f710 legacy=0x7fe9181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:27.819 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.815+0000 7fe9200c8640 1 -- 172.21.15.156:0/178517110 shutdown_connections 2024-03-25T18:12:27.819 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.815+0000 7fe9200c8640 1 -- 172.21.15.156:0/178517110 >> 172.21.15.156:0/178517110 conn(0x7fe9180b7ea0 msgr2=0x7fe9180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:27.819 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.815+0000 7fe9200c8640 1 -- 172.21.15.156:0/178517110 shutdown_connections 2024-03-25T18:12:27.819 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.815+0000 7fe9200c8640 1 -- 172.21.15.156:0/178517110 wait complete. 2024-03-25T18:12:27.819 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.816+0000 7fe9200c8640 10 monclient: build_initial_monmap 2024-03-25T18:12:27.819 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.816+0000 7fe9200c8640 10 monclient: monmap: 2024-03-25T18:12:27.819 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:12:27.819 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:27.819 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:12:27.819 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:12:27.819 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:27.819 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:12:27.819 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:12:27.820 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:12:27.820 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:12:27.820 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:27.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.817+0000 7fe9200c8640 1 librados: starting msgr at 2024-03-25T18:12:27.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.817+0000 7fe9200c8640 1 librados: starting objecter 2024-03-25T18:12:27.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.817+0000 7fe9200c8640 1 Processor -- start 2024-03-25T18:12:27.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.817+0000 7fe9200c8640 1 -- start start 2024-03-25T18:12:27.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.817+0000 7fe9200c8640 1 librados: setting wanted keys 2024-03-25T18:12:27.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.817+0000 7fe9200c8640 1 librados: calling monclient init 2024-03-25T18:12:27.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.817+0000 7fe9200c8640 10 monclient: init 2024-03-25T18:12:27.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.817+0000 7fe9200c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:27.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.817+0000 7fe9200c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:12:27.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.817+0000 7fe9200c8640 10 monclient(hunting): picked mon.noname-b con 0x7fe9180a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:12:27.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.817+0000 7fe9200c8640 10 monclient(hunting): picked mon.noname-c con 0x7fe91813f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:12:27.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.817+0000 7fe9200c8640 10 monclient(hunting): picked mon.noname-a con 0x7fe9181430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:12:27.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.817+0000 7fe9200c8640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9181359a0 con 0x7fe9181430c0 2024-03-25T18:12:27.821 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.817+0000 7fe9200c8640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9181319b0 con 0x7fe9180a98f0 2024-03-25T18:12:27.821 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.817+0000 7fe9200c8640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9180b7bb0 con 0x7fe91813f710 2024-03-25T18:12:27.821 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.817+0000 7fe9200c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:27.821 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.817+0000 7fe9200c8640 10 monclient(hunting): authenticate will time out at 7229.128734s 2024-03-25T18:12:27.823 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.822+0000 7fe91d663640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7fe91813f710 0x7fe918130b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:50988/0 (socket says 172.21.15.156:50988) 2024-03-25T18:12:27.823 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.822+0000 7fe91d663640 1 -- 172.21.15.156:0/1424363349 learned_addr learned my addr 172.21.15.156:0/1424363349 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:12:27.824 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.824+0000 7fe90e7fc640 10 client.?.objecter ms_handle_connect 0x7fe91813f710 2024-03-25T18:12:27.825 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.824+0000 7fe90e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:27.838 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.837+0000 7fe90e7fc640 10 client.?.objecter ms_handle_connect 0x7fe9180a98f0 2024-03-25T18:12:27.838 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.837+0000 7fe90e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:27.841 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.840+0000 7fe90e7fc640 1 -- 172.21.15.156:0/1424363349 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3932210059 0 0) 0x7fe9180b7bb0 con 0x7fe91813f710 2024-03-25T18:12:27.841 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.840+0000 7fe90e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:27.841 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.840+0000 7fe90e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:27.841 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.840+0000 7fe90e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:27.841 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.840+0000 7fe90e7fc640 10 monclient(hunting): my global_id is 4418 2024-03-25T18:12:27.841 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.840+0000 7fe90e7fc640 1 -- 172.21.15.156:0/1424363349 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8fc004eb0 con 0x7fe91813f710 2024-03-25T18:12:27.850 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.850+0000 7fe90e7fc640 10 client.?.objecter ms_handle_connect 0x7fe9181430c0 2024-03-25T18:12:27.850 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.850+0000 7fe90e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:27.853 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.853+0000 7fe90e7fc640 1 -- 172.21.15.156:0/1424363349 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724863654 0 0) 0x7fe9181319b0 con 0x7fe9180a98f0 2024-03-25T18:12:27.854 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.853+0000 7fe90e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:27.854 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.853+0000 7fe90e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:27.854 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.853+0000 7fe90e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:27.854 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.853+0000 7fe90e7fc640 10 monclient(hunting): my global_id is 4411 2024-03-25T18:12:27.854 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.853+0000 7fe90e7fc640 1 -- 172.21.15.156:0/1424363349 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9180b7bb0 con 0x7fe9180a98f0 2024-03-25T18:12:27.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.857+0000 7fe90e7fc640 1 -- 172.21.15.156:0/1424363349 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792424012 0 0) 0x7fe9181359a0 con 0x7fe9181430c0 2024-03-25T18:12:27.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.857+0000 7fe90e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:27.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.857+0000 7fe90e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:27.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.857+0000 7fe90e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:27.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.857+0000 7fe90e7fc640 10 monclient(hunting): my global_id is 4410 2024-03-25T18:12:27.859 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.857+0000 7fe90e7fc640 1 -- 172.21.15.156:0/1424363349 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9181319b0 con 0x7fe9181430c0 2024-03-25T18:12:27.872 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.871+0000 7fe90e7fc640 1 -- 172.21.15.156:0/1424363349 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3907567051 0 0) 0x7fe9180b7bb0 con 0x7fe9180a98f0 2024-03-25T18:12:27.872 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.871+0000 7fe90e7fc640 1 -- 172.21.15.156:0/1424363349 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9181359a0 con 0x7fe9180a98f0 2024-03-25T18:12:27.872 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.871+0000 7fe90e7fc640 1 -- 172.21.15.156:0/1424363349 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fe9140018b0 con 0x7fe9180a98f0 2024-03-25T18:12:27.872 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.871+0000 7fe90e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:27.873 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.871+0000 7fe90e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:27.873 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.871+0000 7fe90e7fc640 10 monclient(hunting): dump: 2024-03-25T18:12:27.873 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:27.873 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:27.873 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:27.873 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:27.873 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:27.873 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:27.873 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:27.873 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:27.873 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:27.873 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:27.873 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.871+0000 7fe90e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:27.877 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.876+0000 7fe90e7fc640 1 -- 172.21.15.156:0/1424363349 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1724172226 0 0) 0x7fe8fc004eb0 con 0x7fe91813f710 2024-03-25T18:12:27.877 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.876+0000 7fe90e7fc640 1 -- 172.21.15.156:0/1424363349 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9180b7bb0 con 0x7fe91813f710 2024-03-25T18:12:27.877 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.876+0000 7fe90e7fc640 1 -- 172.21.15.156:0/1424363349 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fe908001b70 con 0x7fe91813f710 2024-03-25T18:12:27.877 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.876+0000 7fe90e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:27.878 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.876+0000 7fe90e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:27.878 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.876+0000 7fe90e7fc640 10 monclient(hunting): dump: 2024-03-25T18:12:27.878 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:27.878 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:27.878 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:27.878 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:27.878 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:27.878 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:27.878 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:27.878 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:27.878 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:27.878 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:27.888 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.887+0000 7fe90e7fc640 1 -- 172.21.15.156:0/1424363349 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2719932845 0 0) 0x7fe9181359a0 con 0x7fe9180a98f0 2024-03-25T18:12:27.888 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.887+0000 7fe90e7fc640 1 -- 172.21.15.156:0/1424363349 >> v1:172.21.15.156:6790/0 conn(0x7fe91813f710 legacy=0x7fe918130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:27.889 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.887+0000 7fe90e7fc640 1 -- 172.21.15.156:0/1424363349 >> v1:172.21.15.156:6789/0 conn(0x7fe9181430c0 legacy=0x7fe9181312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:27.889 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.887+0000 7fe90e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:12:27.889 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.887+0000 7fe90e7fc640 1 monclient: found mon.b 2024-03-25T18:12:27.889 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.887+0000 7fe90e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:27.889 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.887+0000 7fe90e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:12:27.889 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.887+0000 7fe90e7fc640 1 -- 172.21.15.156:0/1424363349 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe918135070 con 0x7fe9180a98f0 2024-03-25T18:12:27.889 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.887+0000 7fe90e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:12:27.889 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.887+0000 7fe90e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:12:27.889 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.887+0000 7fe90e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:27.889 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.887+0000 7fe9200c8640 5 monclient: authenticate success, global_id 4411 2024-03-25T18:12:27.889 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.887+0000 7fe9200c8640 10 monclient: _renew_subs 2024-03-25T18:12:27.889 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.887+0000 7fe9200c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:12:27.889 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.887+0000 7fe9200c8640 1 -- 172.21.15.156:0/1424363349 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe91813b400 con 0x7fe9180a98f0 2024-03-25T18:12:27.889 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.887+0000 7fe9200c8640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:27.890 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.887+0000 7fe9200c8640 10 monclient: _renew_subs 2024-03-25T18:12:27.890 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.887+0000 7fe9200c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:12:27.890 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.887+0000 7fe9200c8640 1 -- 172.21.15.156:0/1424363349 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9180b75a0 con 0x7fe9180a98f0 2024-03-25T18:12:27.890 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.887+0000 7fe9200c8640 1 librados: init done 2024-03-25T18:12:27.890 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.889+0000 7fe9200c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:27.890 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.889+0000 7fe9200c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:27.891 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.889+0000 7fe9200c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:12:27.891 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.889+0000 7fe9200c8640 1 -- 172.21.15.156:0/1424363349 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe91814f320 con 0x7fe9180a98f0 2024-03-25T18:12:27.898 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.897+0000 7fe90e7fc640 1 -- 172.21.15.156:0/1424363349 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe914003610 con 0x7fe9180a98f0 2024-03-25T18:12:27.898 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.897+0000 7fe90e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:27.898 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:27.897+0000 7fe90ffff640 4 set_mon_vals no callback set 2024-03-25T18:12:28.176 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.175+0000 7fe90e7fc640 1 -- 172.21.15.156:0/1424363349 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fe914003a60 con 0x7fe9180a98f0 2024-03-25T18:12:28.176 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.175+0000 7fe90e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:28.176 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.175+0000 7fe90e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:28.176 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.175+0000 7fe90e7fc640 10 monclient: dump: 2024-03-25T18:12:28.176 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:28.176 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:28.176 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:28.177 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:28.177 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:28.177 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:28.177 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:28.177 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:28.177 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:28.177 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:28.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.175+0000 7fe90e7fc640 1 -- 172.21.15.156:0/1424363349 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7fe914003c40 con 0x7fe9180a98f0 2024-03-25T18:12:28.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.176+0000 7fe91d663640 10 monclient: get_auth_request con 0x7fe8fc06dec0 auth_method 0 2024-03-25T18:12:28.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.176+0000 7fe90e7fc640 10 client.4411.objecter ms_handle_connect 0x7fe8fc06dec0 2024-03-25T18:12:28.179 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.179+0000 7fe90e7fc640 1 -- 172.21.15.156:0/1424363349 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7fe914053620 con 0x7fe9180a98f0 2024-03-25T18:12:28.179 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.179+0000 7fe90e7fc640 10 client.4411.objecter ms_dispatch 0x7fe918000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:28.180 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.179+0000 7fe90e7fc640 3 client.4411.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:28.180 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.179+0000 7fe90e7fc640 3 client.4411.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:28.180 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.179+0000 7fe90e7fc640 20 client.4411.objecter dump_active .. 0 homeless 2024-03-25T18:12:28.182 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.181+0000 7fe90e7fc640 1 -- 172.21.15.156:0/1424363349 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe91404f130 con 0x7fe9180a98f0 2024-03-25T18:12:28.182 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.181+0000 7fe90e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:28.182 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.181+0000 7fe90e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:28.428 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.427+0000 7fe9200c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:28.428 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.427+0000 7fe9200c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:28.428 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.427+0000 7fe9200c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:12:28.428 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.427+0000 7fe9200c8640 1 -- 172.21.15.156:0/1424363349 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe918074cc0 con 0x7fe9180a98f0 2024-03-25T18:12:28.480 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.479+0000 7fe90e7fc640 1 -- 172.21.15.156:0/1424363349 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7fe914051060 con 0x7fe9180a98f0 2024-03-25T18:12:28.480 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.479+0000 7fe90e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:28.480 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.479+0000 7fe90e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:28.480 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:12:28.480 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:28.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.481+0000 7fe9200c8640 10 librados: watch_flush enter 2024-03-25T18:12:28.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.481+0000 7fe9200c8640 10 librados: watch_flush exit 2024-03-25T18:12:28.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.481+0000 7fe9200c8640 20 client.4411.objecter shutdown clearing up homeless session... 2024-03-25T18:12:28.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.481+0000 7fe9200c8640 10 client.4411.objecter successfully canceled tick 2024-03-25T18:12:28.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.481+0000 7fe9200c8640 1 -- 172.21.15.156:0/1424363349 >> v1:172.21.15.194:6800/2946427713 conn(0x7fe8fc06dec0 legacy=0x7fe8fc070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:28.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.481+0000 7fe9200c8640 10 monclient: shutdown 2024-03-25T18:12:28.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.481+0000 7fe9200c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:28.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.481+0000 7fe9200c8640 1 -- 172.21.15.156:0/1424363349 >> v1:172.21.15.194:6789/0 conn(0x7fe9180a98f0 legacy=0x7fe9181d7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:28.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.481+0000 7fe9200c8640 1 -- 172.21.15.156:0/1424363349 shutdown_connections 2024-03-25T18:12:28.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.481+0000 7fe9200c8640 1 -- 172.21.15.156:0/1424363349 >> 172.21.15.156:0/1424363349 conn(0x7fe9180b7ea0 msgr2=0x7fe918074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:28.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.481+0000 7fe9200c8640 1 -- 172.21.15.156:0/1424363349 shutdown_connections 2024-03-25T18:12:28.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.481+0000 7fe9200c8640 1 -- 172.21.15.156:0/1424363349 wait complete. 2024-03-25T18:12:28.484 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:28.481+0000 7fe9200c8640 1 librados: shutdown 2024-03-25T18:12:28.493 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:35.300 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:35.417 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.416+0000 7fbd8dd13640 10 monclient: get_monmap_and_config 2024-03-25T18:12:35.417 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.416+0000 7fbd8dd13640 10 monclient: build_initial_monmap 2024-03-25T18:12:35.417 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.416+0000 7fbd8dd13640 10 monclient: monmap: 2024-03-25T18:12:35.417 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:12:35.417 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:35.417 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:12:35.417236+0000 2024-03-25T18:12:35.417 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:12:35.417236+0000 2024-03-25T18:12:35.417 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:35.417 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:35.418 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:12:35.418 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:12:35.418 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:12:35.418 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:35.419 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.417+0000 7fbd8dd13640 1 Processor -- start 2024-03-25T18:12:35.419 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.417+0000 7fbd8dd13640 1 -- start start 2024-03-25T18:12:35.419 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.417+0000 7fbd8dd13640 10 monclient: init 2024-03-25T18:12:35.419 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.417+0000 7fbd8dd13640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:35.419 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.417+0000 7fbd8dd13640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:12:35.419 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.417+0000 7fbd8dd13640 10 monclient(hunting): picked mon.noname-b con 0x7fbd880a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:12:35.419 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.417+0000 7fbd8dd13640 10 monclient(hunting): picked mon.noname-a con 0x7fbd8813f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:12:35.419 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.417+0000 7fbd8dd13640 10 monclient(hunting): picked mon.noname-c con 0x7fbd881430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:12:35.419 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.417+0000 7fbd8dd13640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd88142520 con 0x7fbd8813f710 2024-03-25T18:12:35.419 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.417+0000 7fbd8dd13640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd880b28a0 con 0x7fbd881430c0 2024-03-25T18:12:35.419 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.418+0000 7fbd8dd13640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd880b2a30 con 0x7fbd880a98f0 2024-03-25T18:12:35.419 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.418+0000 7fbd8dd13640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:35.419 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.418+0000 7fbd8dd13640 10 monclient(hunting): authenticate will time out at 7236.728809s 2024-03-25T18:12:35.428 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.428+0000 7fbd87fff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fbd881430c0 0x7fbd8814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:40552/0 (socket says 172.21.15.156:40552) 2024-03-25T18:12:35.429 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.428+0000 7fbd87fff640 1 -- 172.21.15.156:0/958303009 learned_addr learned my addr 172.21.15.156:0/958303009 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:12:35.441 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.440+0000 7fbd867fc640 1 -- 172.21.15.156:0/958303009 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 938946536 0 0) 0x7fbd88142520 con 0x7fbd8813f710 2024-03-25T18:12:35.441 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.440+0000 7fbd867fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:35.441 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.440+0000 7fbd867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:35.441 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.440+0000 7fbd867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:35.441 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.440+0000 7fbd867fc640 10 monclient(hunting): my global_id is 4413 2024-03-25T18:12:35.441 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.440+0000 7fbd867fc640 1 -- 172.21.15.156:0/958303009 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd70003520 con 0x7fbd8813f710 2024-03-25T18:12:35.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.442+0000 7fbd867fc640 1 -- 172.21.15.156:0/958303009 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2083332790 0 0) 0x7fbd880b28a0 con 0x7fbd881430c0 2024-03-25T18:12:35.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.442+0000 7fbd867fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:35.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.442+0000 7fbd867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:35.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.442+0000 7fbd867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:35.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.442+0000 7fbd867fc640 10 monclient(hunting): my global_id is 4414 2024-03-25T18:12:35.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.442+0000 7fbd867fc640 1 -- 172.21.15.156:0/958303009 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd88142520 con 0x7fbd881430c0 2024-03-25T18:12:35.449 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.448+0000 7fbd867fc640 1 -- 172.21.15.156:0/958303009 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981191060 0 0) 0x7fbd880b2a30 con 0x7fbd880a98f0 2024-03-25T18:12:35.449 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.448+0000 7fbd867fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:35.449 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.448+0000 7fbd867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:35.450 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.448+0000 7fbd867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:35.450 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.448+0000 7fbd867fc640 10 monclient(hunting): my global_id is 4421 2024-03-25T18:12:35.450 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.448+0000 7fbd867fc640 1 -- 172.21.15.156:0/958303009 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd880b28a0 con 0x7fbd880a98f0 2024-03-25T18:12:35.451 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.449+0000 7fbd867fc640 1 -- 172.21.15.156:0/958303009 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3382582686 0 0) 0x7fbd70003520 con 0x7fbd8813f710 2024-03-25T18:12:35.451 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.449+0000 7fbd867fc640 1 -- 172.21.15.156:0/958303009 >> v1:172.21.15.156:6790/0 conn(0x7fbd880a98f0 legacy=0x7fbd880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:35.451 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.449+0000 7fbd867fc640 1 -- 172.21.15.156:0/958303009 >> v1:172.21.15.194:6789/0 conn(0x7fbd881430c0 legacy=0x7fbd8814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:35.451 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.449+0000 7fbd867fc640 10 monclient: _finish_hunting 0 2024-03-25T18:12:35.451 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.449+0000 7fbd867fc640 1 monclient: found mon.noname-a 2024-03-25T18:12:35.451 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.449+0000 7fbd867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:35.451 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.449+0000 7fbd867fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:12:35.451 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.449+0000 7fbd867fc640 1 -- 172.21.15.156:0/958303009 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd880b44c0 con 0x7fbd8813f710 2024-03-25T18:12:35.451 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.449+0000 7fbd867fc640 10 monclient: _finish_auth 0 2024-03-25T18:12:35.451 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.449+0000 7fbd867fc640 10 monclient: _check_auth_tickets 2024-03-25T18:12:35.451 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.449+0000 7fbd867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:35.451 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.449+0000 7fbd867fc640 1 -- 172.21.15.156:0/958303009 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fbd7c001690 con 0x7fbd8813f710 2024-03-25T18:12:35.452 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.449+0000 7fbd8dd13640 5 monclient: authenticate success, global_id 4413 2024-03-25T18:12:35.452 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.449+0000 7fbd8dd13640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:35.452 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.449+0000 7fbd867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:35.452 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.449+0000 7fbd867fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:35.452 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.449+0000 7fbd867fc640 10 monclient: dump: 2024-03-25T18:12:35.452 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:35.452 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:35.452 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:35.452 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:35.452 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:35.452 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:35.452 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:35.452 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:35.452 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:35.452 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:35.453 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.449+0000 7fbd8dd13640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:35.455 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.454+0000 7fbd867fc640 1 -- 172.21.15.156:0/958303009 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd7c0017f0 con 0x7fbd8813f710 2024-03-25T18:12:35.455 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.454+0000 7fbd867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:35.455 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.454+0000 7fbd867fc640 1 -- 172.21.15.156:0/958303009 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fbd7c001c00 con 0x7fbd8813f710 2024-03-25T18:12:35.455 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.454+0000 7fbd8dd13640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:35.455 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.454+0000 7fbd8dd13640 4 set_mon_vals no callback set 2024-03-25T18:12:35.455 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.454+0000 7fbd867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:35.455 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.454+0000 7fbd867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:35.455 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.454+0000 7fbd867fc640 10 monclient: dump: 2024-03-25T18:12:35.455 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:35.455 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:35.455 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:35.455 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:35.456 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:35.456 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:35.456 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:35.456 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:35.456 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:35.456 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:35.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.454+0000 7fbd8dd13640 10 monclient: shutdown 2024-03-25T18:12:35.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.454+0000 7fbd8dd13640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:35.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.454+0000 7fbd8dd13640 1 -- 172.21.15.156:0/958303009 >> v1:172.21.15.156:6789/0 conn(0x7fbd8813f710 legacy=0x7fbd881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:35.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.454+0000 7fbd8dd13640 1 -- 172.21.15.156:0/958303009 shutdown_connections 2024-03-25T18:12:35.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.454+0000 7fbd8dd13640 1 -- 172.21.15.156:0/958303009 >> 172.21.15.156:0/958303009 conn(0x7fbd880b7ea0 msgr2=0x7fbd880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:35.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.455+0000 7fbd8dd13640 1 -- 172.21.15.156:0/958303009 shutdown_connections 2024-03-25T18:12:35.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.455+0000 7fbd8dd13640 1 -- 172.21.15.156:0/958303009 wait complete. 2024-03-25T18:12:35.459 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.455+0000 7fbd8dd13640 10 monclient: build_initial_monmap 2024-03-25T18:12:35.459 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.455+0000 7fbd8dd13640 10 monclient: monmap: 2024-03-25T18:12:35.459 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:12:35.459 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:35.459 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:12:35.459 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:12:35.459 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:35.459 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:12:35.459 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:12:35.459 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:12:35.459 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:12:35.459 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:35.459 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.455+0000 7fbd8dd13640 1 librados: starting msgr at 2024-03-25T18:12:35.459 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.455+0000 7fbd8dd13640 1 librados: starting objecter 2024-03-25T18:12:35.459 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.456+0000 7fbd8dd13640 1 Processor -- start 2024-03-25T18:12:35.459 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.456+0000 7fbd8dd13640 1 -- start start 2024-03-25T18:12:35.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.456+0000 7fbd8dd13640 1 librados: setting wanted keys 2024-03-25T18:12:35.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.456+0000 7fbd8dd13640 1 librados: calling monclient init 2024-03-25T18:12:35.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.456+0000 7fbd8dd13640 10 monclient: init 2024-03-25T18:12:35.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.456+0000 7fbd8dd13640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:35.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.456+0000 7fbd8dd13640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:12:35.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.456+0000 7fbd8dd13640 10 monclient(hunting): picked mon.noname-b con 0x7fbd880a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:12:35.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.456+0000 7fbd8dd13640 10 monclient(hunting): picked mon.noname-c con 0x7fbd8813f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:12:35.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.456+0000 7fbd8dd13640 10 monclient(hunting): picked mon.noname-a con 0x7fbd881430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:12:35.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.456+0000 7fbd8dd13640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd88137f10 con 0x7fbd881430c0 2024-03-25T18:12:35.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.457+0000 7fbd8dd13640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd881dc070 con 0x7fbd880a98f0 2024-03-25T18:12:35.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.457+0000 7fbd8dd13640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd88137100 con 0x7fbd8813f710 2024-03-25T18:12:35.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.457+0000 7fbd8dd13640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:35.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.457+0000 7fbd8dd13640 10 monclient(hunting): authenticate will time out at 7236.767822s 2024-03-25T18:12:35.462 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.461+0000 7fbd87fff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fbd881430c0 0x7fbd881db960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:42318/0 (socket says 172.21.15.156:42318) 2024-03-25T18:12:35.462 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.461+0000 7fbd87fff640 1 -- 172.21.15.156:0/686897575 learned_addr learned my addr 172.21.15.156:0/686897575 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:12:35.464 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.463+0000 7fbd6ffff640 10 client.?.objecter ms_handle_connect 0x7fbd881430c0 2024-03-25T18:12:35.464 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.464+0000 7fbd6ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:35.472 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.471+0000 7fbd6ffff640 1 -- 172.21.15.156:0/686897575 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296317668 0 0) 0x7fbd88137f10 con 0x7fbd881430c0 2024-03-25T18:12:35.472 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.471+0000 7fbd6ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:35.472 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.471+0000 7fbd6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:35.472 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.471+0000 7fbd6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:35.472 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.471+0000 7fbd6ffff640 10 monclient(hunting): my global_id is 4416 2024-03-25T18:12:35.472 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.471+0000 7fbd6ffff640 1 -- 172.21.15.156:0/686897575 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd74004eb0 con 0x7fbd881430c0 2024-03-25T18:12:35.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.482+0000 7fbd6ffff640 10 client.?.objecter ms_handle_connect 0x7fbd880a98f0 2024-03-25T18:12:35.483 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.482+0000 7fbd6ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:35.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.486+0000 7fbd6ffff640 1 -- 172.21.15.156:0/686897575 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3797406800 0 0) 0x7fbd74004eb0 con 0x7fbd881430c0 2024-03-25T18:12:35.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.486+0000 7fbd6ffff640 1 -- 172.21.15.156:0/686897575 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd88137f10 con 0x7fbd881430c0 2024-03-25T18:12:35.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.486+0000 7fbd6ffff640 1 -- 172.21.15.156:0/686897575 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fbd80001c70 con 0x7fbd881430c0 2024-03-25T18:12:35.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.486+0000 7fbd6ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:35.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.486+0000 7fbd6ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:35.488 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.486+0000 7fbd6ffff640 10 monclient(hunting): dump: 2024-03-25T18:12:35.488 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:35.488 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:35.488 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:35.488 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:35.488 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:35.488 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:35.488 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:35.488 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:35.488 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:35.488 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:35.488 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.486+0000 7fbd6ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:35.489 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.487+0000 7fbd6ffff640 10 client.?.objecter ms_handle_connect 0x7fbd8813f710 2024-03-25T18:12:35.489 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.487+0000 7fbd6ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:35.500 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.499+0000 7fbd6ffff640 1 -- 172.21.15.156:0/686897575 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 838623987 0 0) 0x7fbd88137100 con 0x7fbd8813f710 2024-03-25T18:12:35.500 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.499+0000 7fbd6ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:35.500 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.499+0000 7fbd6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:35.500 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.499+0000 7fbd6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:35.500 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.499+0000 7fbd6ffff640 10 monclient(hunting): my global_id is 4424 2024-03-25T18:12:35.500 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.499+0000 7fbd6ffff640 1 -- 172.21.15.156:0/686897575 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd74004eb0 con 0x7fbd8813f710 2024-03-25T18:12:35.501 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.500+0000 7fbd6ffff640 1 -- 172.21.15.156:0/686897575 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4146849191 0 0) 0x7fbd881dc070 con 0x7fbd880a98f0 2024-03-25T18:12:35.501 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.500+0000 7fbd6ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:35.501 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.500+0000 7fbd6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:35.501 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.500+0000 7fbd6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:35.501 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.500+0000 7fbd6ffff640 10 monclient(hunting): my global_id is 4417 2024-03-25T18:12:35.501 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.500+0000 7fbd6ffff640 1 -- 172.21.15.156:0/686897575 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd88137100 con 0x7fbd880a98f0 2024-03-25T18:12:35.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.507+0000 7fbd6ffff640 1 -- 172.21.15.156:0/686897575 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3159273438 0 0) 0x7fbd88137f10 con 0x7fbd881430c0 2024-03-25T18:12:35.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.507+0000 7fbd6ffff640 1 -- 172.21.15.156:0/686897575 >> v1:172.21.15.156:6790/0 conn(0x7fbd8813f710 legacy=0x7fbd881db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:35.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.507+0000 7fbd6ffff640 1 -- 172.21.15.156:0/686897575 >> v1:172.21.15.194:6789/0 conn(0x7fbd880a98f0 legacy=0x7fbd88137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:35.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.507+0000 7fbd6ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:12:35.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.507+0000 7fbd6ffff640 1 monclient: found mon.a 2024-03-25T18:12:35.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.507+0000 7fbd6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:35.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.507+0000 7fbd6ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:12:35.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.507+0000 7fbd6ffff640 1 -- 172.21.15.156:0/686897575 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd881373e0 con 0x7fbd881430c0 2024-03-25T18:12:35.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.507+0000 7fbd6ffff640 10 monclient: _finish_auth 0 2024-03-25T18:12:35.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.507+0000 7fbd6ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:12:35.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.507+0000 7fbd6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:35.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.507+0000 7fbd8dd13640 5 monclient: authenticate success, global_id 4416 2024-03-25T18:12:35.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.507+0000 7fbd8dd13640 10 monclient: _renew_subs 2024-03-25T18:12:35.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.507+0000 7fbd8dd13640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:12:35.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.507+0000 7fbd8dd13640 1 -- 172.21.15.156:0/686897575 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd881358e0 con 0x7fbd881430c0 2024-03-25T18:12:35.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.507+0000 7fbd8dd13640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:35.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.507+0000 7fbd8dd13640 10 monclient: _renew_subs 2024-03-25T18:12:35.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.507+0000 7fbd8dd13640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:12:35.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.507+0000 7fbd8dd13640 1 -- 172.21.15.156:0/686897575 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd88136150 con 0x7fbd881430c0 2024-03-25T18:12:35.510 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.507+0000 7fbd8dd13640 1 librados: init done 2024-03-25T18:12:35.510 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.508+0000 7fbd8dd13640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:35.510 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.508+0000 7fbd8dd13640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:35.510 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.508+0000 7fbd8dd13640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:12:35.510 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.508+0000 7fbd8dd13640 1 -- 172.21.15.156:0/686897575 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd880627b0 con 0x7fbd881430c0 2024-03-25T18:12:35.512 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.511+0000 7fbd6ffff640 1 -- 172.21.15.156:0/686897575 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd80004b60 con 0x7fbd881430c0 2024-03-25T18:12:35.512 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.511+0000 7fbd6ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:35.512 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.511+0000 7fbd85ffb640 4 set_mon_vals no callback set 2024-03-25T18:12:35.805 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.804+0000 7fbd6ffff640 1 -- 172.21.15.156:0/686897575 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fbd80004e70 con 0x7fbd881430c0 2024-03-25T18:12:35.806 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.804+0000 7fbd6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:35.806 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.804+0000 7fbd6ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:35.806 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.804+0000 7fbd6ffff640 10 monclient: dump: 2024-03-25T18:12:35.806 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:35.806 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:35.806 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:35.806 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:35.806 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:35.806 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:35.806 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:35.806 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:35.806 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:35.807 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:35.807 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.805+0000 7fbd6ffff640 1 -- 172.21.15.156:0/686897575 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7fbd8001ae40 con 0x7fbd881430c0 2024-03-25T18:12:35.807 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.806+0000 7fbd877fe640 10 monclient: get_auth_request con 0x7fbd7406de10 auth_method 0 2024-03-25T18:12:35.807 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.806+0000 7fbd6ffff640 10 client.4416.objecter ms_handle_connect 0x7fbd7406de10 2024-03-25T18:12:35.811 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.810+0000 7fbd6ffff640 1 -- 172.21.15.156:0/686897575 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7fbd80054bf0 con 0x7fbd881430c0 2024-03-25T18:12:35.811 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.810+0000 7fbd6ffff640 10 client.4416.objecter ms_dispatch 0x7fbd88000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:35.811 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.810+0000 7fbd6ffff640 3 client.4416.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:35.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.810+0000 7fbd6ffff640 3 client.4416.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:35.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.810+0000 7fbd6ffff640 20 client.4416.objecter dump_active .. 0 homeless 2024-03-25T18:12:35.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.810+0000 7fbd6ffff640 1 -- 172.21.15.156:0/686897575 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbd80050530 con 0x7fbd881430c0 2024-03-25T18:12:35.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.810+0000 7fbd6ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:35.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:35.810+0000 7fbd6ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:36.059 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:36.058+0000 7fbd8dd13640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:36.059 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:36.058+0000 7fbd8dd13640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:36.059 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:36.058+0000 7fbd8dd13640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:12:36.059 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:36.058+0000 7fbd8dd13640 1 -- 172.21.15.156:0/686897575 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd88135a70 con 0x7fbd881430c0 2024-03-25T18:12:36.084 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:36.083+0000 7fbd6ffff640 1 -- 172.21.15.156:0/686897575 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7fbd80052630 con 0x7fbd881430c0 2024-03-25T18:12:36.084 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:36.083+0000 7fbd6ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:36.084 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:36.083+0000 7fbd6ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:36.084 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:12:36.084 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:36.087 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:36.086+0000 7fbd8dd13640 10 librados: watch_flush enter 2024-03-25T18:12:36.087 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:36.086+0000 7fbd8dd13640 10 librados: watch_flush exit 2024-03-25T18:12:36.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:36.086+0000 7fbd8dd13640 20 client.4416.objecter shutdown clearing up homeless session... 2024-03-25T18:12:36.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:36.086+0000 7fbd8dd13640 10 client.4416.objecter successfully canceled tick 2024-03-25T18:12:36.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:36.086+0000 7fbd8dd13640 1 -- 172.21.15.156:0/686897575 >> v1:172.21.15.194:6800/2946427713 conn(0x7fbd7406de10 legacy=0x7fbd740702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:36.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:36.086+0000 7fbd8dd13640 10 monclient: shutdown 2024-03-25T18:12:36.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:36.086+0000 7fbd8dd13640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:36.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:36.086+0000 7fbd8dd13640 1 -- 172.21.15.156:0/686897575 >> v1:172.21.15.156:6789/0 conn(0x7fbd881430c0 legacy=0x7fbd881db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:36.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:36.086+0000 7fbd8dd13640 1 -- 172.21.15.156:0/686897575 shutdown_connections 2024-03-25T18:12:36.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:36.086+0000 7fbd8dd13640 1 -- 172.21.15.156:0/686897575 >> 172.21.15.156:0/686897575 conn(0x7fbd880b7ea0 msgr2=0x7fbd8814ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:36.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:36.086+0000 7fbd8dd13640 1 -- 172.21.15.156:0/686897575 shutdown_connections 2024-03-25T18:12:36.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:36.086+0000 7fbd8dd13640 1 -- 172.21.15.156:0/686897575 wait complete. 2024-03-25T18:12:36.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:36.086+0000 7fbd8dd13640 1 librados: shutdown 2024-03-25T18:12:36.099 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:42.906 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:43.020 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.019+0000 7fc0dce74640 10 monclient: get_monmap_and_config 2024-03-25T18:12:43.020 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.019+0000 7fc0dce74640 10 monclient: build_initial_monmap 2024-03-25T18:12:43.020 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.019+0000 7fc0dce74640 10 monclient: monmap: 2024-03-25T18:12:43.021 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:12:43.021 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:43.021 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:12:43.020637+0000 2024-03-25T18:12:43.021 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:12:43.020637+0000 2024-03-25T18:12:43.021 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:43.021 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:43.021 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:12:43.021 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:12:43.021 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:12:43.021 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:43.022 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.020+0000 7fc0dce74640 1 Processor -- start 2024-03-25T18:12:43.022 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.020+0000 7fc0dce74640 1 -- start start 2024-03-25T18:12:43.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.020+0000 7fc0dce74640 10 monclient: init 2024-03-25T18:12:43.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.021+0000 7fc0dce74640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:43.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.021+0000 7fc0dce74640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:12:43.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.021+0000 7fc0dce74640 10 monclient(hunting): picked mon.noname-b con 0x7fc0d80a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:12:43.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.021+0000 7fc0dce74640 10 monclient(hunting): picked mon.noname-a con 0x7fc0d813f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:12:43.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.021+0000 7fc0dce74640 10 monclient(hunting): picked mon.noname-c con 0x7fc0d81430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:12:43.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.021+0000 7fc0dce74640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d8142520 con 0x7fc0d813f710 2024-03-25T18:12:43.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.021+0000 7fc0dce74640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d80b28a0 con 0x7fc0d81430c0 2024-03-25T18:12:43.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.021+0000 7fc0dce74640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d80b2a30 con 0x7fc0d80a98f0 2024-03-25T18:12:43.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.021+0000 7fc0dce74640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:43.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.021+0000 7fc0dce74640 10 monclient(hunting): authenticate will time out at 7244.332202s 2024-03-25T18:12:43.034 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.033+0000 7fc0d5d9b640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fc0d813f710 0x7fc0d81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:49178/0 (socket says 172.21.15.156:49178) 2024-03-25T18:12:43.034 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.033+0000 7fc0d5d9b640 1 -- 172.21.15.156:0/723957982 learned_addr learned my addr 172.21.15.156:0/723957982 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:12:43.046 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.045+0000 7fc0d559a640 1 -- 172.21.15.156:0/723957982 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662526538 0 0) 0x7fc0d8142520 con 0x7fc0d813f710 2024-03-25T18:12:43.046 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.045+0000 7fc0d559a640 10 monclient(hunting): _negotiate 2024-03-25T18:12:43.046 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.045+0000 7fc0d559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:43.046 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.045+0000 7fc0d559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:43.046 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.045+0000 7fc0d559a640 10 monclient(hunting): my global_id is 4419 2024-03-25T18:12:43.046 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.045+0000 7fc0d559a640 1 -- 172.21.15.156:0/723957982 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0c0003520 con 0x7fc0d813f710 2024-03-25T18:12:43.061 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.060+0000 7fc0d559a640 1 -- 172.21.15.156:0/723957982 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3286277471 0 0) 0x7fc0d80b2a30 con 0x7fc0d80a98f0 2024-03-25T18:12:43.061 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.060+0000 7fc0d559a640 10 monclient(hunting): _negotiate 2024-03-25T18:12:43.061 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.060+0000 7fc0d559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:43.061 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.060+0000 7fc0d559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:43.061 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.060+0000 7fc0d559a640 10 monclient(hunting): my global_id is 4427 2024-03-25T18:12:43.061 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.060+0000 7fc0d559a640 1 -- 172.21.15.156:0/723957982 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0d8142520 con 0x7fc0d80a98f0 2024-03-25T18:12:43.064 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.063+0000 7fc0d559a640 1 -- 172.21.15.156:0/723957982 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 25057191 0 0) 0x7fc0d80b28a0 con 0x7fc0d81430c0 2024-03-25T18:12:43.064 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.064+0000 7fc0d559a640 10 monclient(hunting): _negotiate 2024-03-25T18:12:43.065 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.064+0000 7fc0d559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:43.065 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.064+0000 7fc0d559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:43.065 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.064+0000 7fc0d559a640 10 monclient(hunting): my global_id is 4420 2024-03-25T18:12:43.065 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.064+0000 7fc0d559a640 1 -- 172.21.15.156:0/723957982 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0d80b2a30 con 0x7fc0d81430c0 2024-03-25T18:12:43.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.069+0000 7fc0d559a640 1 -- 172.21.15.156:0/723957982 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 324491120 0 0) 0x7fc0c0003520 con 0x7fc0d813f710 2024-03-25T18:12:43.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.069+0000 7fc0d559a640 1 -- 172.21.15.156:0/723957982 >> v1:172.21.15.156:6790/0 conn(0x7fc0d80a98f0 legacy=0x7fc0d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:43.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.069+0000 7fc0d559a640 1 -- 172.21.15.156:0/723957982 >> v1:172.21.15.194:6789/0 conn(0x7fc0d81430c0 legacy=0x7fc0d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:43.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.069+0000 7fc0d559a640 10 monclient: _finish_hunting 0 2024-03-25T18:12:43.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.069+0000 7fc0d559a640 1 monclient: found mon.noname-a 2024-03-25T18:12:43.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.069+0000 7fc0d559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:43.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.070+0000 7fc0d559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:12:43.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.070+0000 7fc0d559a640 1 -- 172.21.15.156:0/723957982 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d80b44c0 con 0x7fc0d813f710 2024-03-25T18:12:43.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.070+0000 7fc0d559a640 10 monclient: _finish_auth 0 2024-03-25T18:12:43.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.070+0000 7fc0d559a640 10 monclient: _check_auth_tickets 2024-03-25T18:12:43.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.070+0000 7fc0d559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:43.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.070+0000 7fc0dce74640 5 monclient: authenticate success, global_id 4419 2024-03-25T18:12:43.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.070+0000 7fc0d559a640 1 -- 172.21.15.156:0/723957982 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc0cc001690 con 0x7fc0d813f710 2024-03-25T18:12:43.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.070+0000 7fc0d559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:43.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.070+0000 7fc0d559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:43.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.070+0000 7fc0d559a640 10 monclient: dump: 2024-03-25T18:12:43.072 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:43.072 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:43.072 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:43.072 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:43.072 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:43.072 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:43.072 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:43.072 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:43.072 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:43.073 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:43.073 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.070+0000 7fc0dce74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:43.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.075+0000 7fc0d559a640 1 -- 172.21.15.156:0/723957982 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0cc0017f0 con 0x7fc0d813f710 2024-03-25T18:12:43.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.075+0000 7fc0d559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:43.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.075+0000 7fc0dce74640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:43.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.075+0000 7fc0dce74640 4 set_mon_vals no callback set 2024-03-25T18:12:43.077 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.076+0000 7fc0dce74640 10 monclient: shutdown 2024-03-25T18:12:43.077 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.076+0000 7fc0dce74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:43.077 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.076+0000 7fc0dce74640 1 -- 172.21.15.156:0/723957982 >> v1:172.21.15.156:6789/0 conn(0x7fc0d813f710 legacy=0x7fc0d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:43.077 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.076+0000 7fc0dce74640 1 -- 172.21.15.156:0/723957982 shutdown_connections 2024-03-25T18:12:43.077 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.076+0000 7fc0dce74640 1 -- 172.21.15.156:0/723957982 >> 172.21.15.156:0/723957982 conn(0x7fc0d80b7ea0 msgr2=0x7fc0d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:43.077 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.076+0000 7fc0dce74640 1 -- 172.21.15.156:0/723957982 shutdown_connections 2024-03-25T18:12:43.077 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.076+0000 7fc0dce74640 1 -- 172.21.15.156:0/723957982 wait complete. 2024-03-25T18:12:43.078 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.077+0000 7fc0dce74640 10 monclient: build_initial_monmap 2024-03-25T18:12:43.078 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.077+0000 7fc0dce74640 10 monclient: monmap: 2024-03-25T18:12:43.078 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:12:43.078 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:43.078 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:12:43.078 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:12:43.078 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:43.078 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:12:43.078 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:12:43.078 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:12:43.078 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:12:43.079 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:43.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.077+0000 7fc0dce74640 1 librados: starting msgr at 2024-03-25T18:12:43.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.077+0000 7fc0dce74640 1 librados: starting objecter 2024-03-25T18:12:43.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.077+0000 7fc0dce74640 1 Processor -- start 2024-03-25T18:12:43.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.078+0000 7fc0dce74640 1 -- start start 2024-03-25T18:12:43.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.078+0000 7fc0dce74640 1 librados: setting wanted keys 2024-03-25T18:12:43.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.078+0000 7fc0dce74640 1 librados: calling monclient init 2024-03-25T18:12:43.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.078+0000 7fc0dce74640 10 monclient: init 2024-03-25T18:12:43.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.078+0000 7fc0dce74640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:43.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.078+0000 7fc0dce74640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:12:43.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.079+0000 7fc0dce74640 10 monclient(hunting): picked mon.noname-a con 0x7fc0d80a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:12:43.081 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.079+0000 7fc0dce74640 10 monclient(hunting): picked mon.noname-b con 0x7fc0d813f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:12:43.081 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.079+0000 7fc0dce74640 10 monclient(hunting): picked mon.noname-c con 0x7fc0d81430c0 addr v1:172.21.15.156:6790/0 2024-03-25T18:12:43.081 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.079+0000 7fc0dce74640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d8136c00 con 0x7fc0d80a98f0 2024-03-25T18:12:43.081 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.079+0000 7fc0dce74640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d813b600 con 0x7fc0d813f710 2024-03-25T18:12:43.081 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.079+0000 7fc0dce74640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d813b260 con 0x7fc0d81430c0 2024-03-25T18:12:43.081 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.079+0000 7fc0dce74640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:43.081 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.079+0000 7fc0dce74640 10 monclient(hunting): authenticate will time out at 7244.390576s 2024-03-25T18:12:43.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.097+0000 7fc0d5d9b640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fc0d813f710 0x7fc0d81db0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:45270/0 (socket says 172.21.15.156:45270) 2024-03-25T18:12:43.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.097+0000 7fc0d5d9b640 1 -- 172.21.15.156:0/3578355352 learned_addr learned my addr 172.21.15.156:0/3578355352 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:12:43.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.100+0000 7fc0c6ffd640 10 client.?.objecter ms_handle_connect 0x7fc0d813f710 2024-03-25T18:12:43.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.100+0000 7fc0c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:43.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.101+0000 7fc0c6ffd640 10 client.?.objecter ms_handle_connect 0x7fc0d80a98f0 2024-03-25T18:12:43.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.101+0000 7fc0c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:43.103 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.102+0000 7fc0c6ffd640 10 client.?.objecter ms_handle_connect 0x7fc0d81430c0 2024-03-25T18:12:43.103 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.102+0000 7fc0c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:43.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.109+0000 7fc0c6ffd640 1 -- 172.21.15.156:0/3578355352 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 862393906 0 0) 0x7fc0d813b600 con 0x7fc0d813f710 2024-03-25T18:12:43.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.109+0000 7fc0c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:43.111 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.109+0000 7fc0c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:43.111 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.109+0000 7fc0c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:43.111 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.109+0000 7fc0c6ffd640 10 monclient(hunting): my global_id is 4423 2024-03-25T18:12:43.111 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.109+0000 7fc0c6ffd640 1 -- 172.21.15.156:0/3578355352 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0b8004eb0 con 0x7fc0d813f710 2024-03-25T18:12:43.118 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.117+0000 7fc0c6ffd640 1 -- 172.21.15.156:0/3578355352 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2186451465 0 0) 0x7fc0d8136c00 con 0x7fc0d80a98f0 2024-03-25T18:12:43.118 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.117+0000 7fc0c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:43.118 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.117+0000 7fc0c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:43.118 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.117+0000 7fc0c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:43.118 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.117+0000 7fc0c6ffd640 10 monclient(hunting): my global_id is 4422 2024-03-25T18:12:43.118 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.117+0000 7fc0c6ffd640 1 -- 172.21.15.156:0/3578355352 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0d813b600 con 0x7fc0d80a98f0 2024-03-25T18:12:43.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.119+0000 7fc0c6ffd640 1 -- 172.21.15.156:0/3578355352 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439143949 0 0) 0x7fc0d813b260 con 0x7fc0d81430c0 2024-03-25T18:12:43.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.119+0000 7fc0c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:43.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.119+0000 7fc0c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:43.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.119+0000 7fc0c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:43.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.119+0000 7fc0c6ffd640 10 monclient(hunting): my global_id is 4430 2024-03-25T18:12:43.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.119+0000 7fc0c6ffd640 1 -- 172.21.15.156:0/3578355352 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0d8136c00 con 0x7fc0d81430c0 2024-03-25T18:12:43.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.127+0000 7fc0c6ffd640 1 -- 172.21.15.156:0/3578355352 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3496049100 0 0) 0x7fc0b8004eb0 con 0x7fc0d813f710 2024-03-25T18:12:43.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.128+0000 7fc0c6ffd640 1 -- 172.21.15.156:0/3578355352 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0d813b260 con 0x7fc0d813f710 2024-03-25T18:12:43.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.128+0000 7fc0c6ffd640 1 -- 172.21.15.156:0/3578355352 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc0cc001ce0 con 0x7fc0d813f710 2024-03-25T18:12:43.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.128+0000 7fc0c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:43.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.128+0000 7fc0c6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:43.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.128+0000 7fc0c6ffd640 10 monclient(hunting): dump: 2024-03-25T18:12:43.129 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:43.129 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:43.129 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:43.129 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:43.129 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:43.130 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:43.130 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:43.130 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:43.130 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:43.130 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:43.130 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.128+0000 7fc0c6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:43.139 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.138+0000 7fc0c6ffd640 1 -- 172.21.15.156:0/3578355352 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 18598989 0 0) 0x7fc0d813b600 con 0x7fc0d80a98f0 2024-03-25T18:12:43.140 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.138+0000 7fc0c6ffd640 1 -- 172.21.15.156:0/3578355352 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0b8004eb0 con 0x7fc0d80a98f0 2024-03-25T18:12:43.140 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.138+0000 7fc0c6ffd640 1 -- 172.21.15.156:0/3578355352 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc0c8001970 con 0x7fc0d80a98f0 2024-03-25T18:12:43.140 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.138+0000 7fc0c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:43.140 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.138+0000 7fc0c6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:43.140 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.138+0000 7fc0c6ffd640 10 monclient(hunting): dump: 2024-03-25T18:12:43.140 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:43.140 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:43.140 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:43.140 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:43.140 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:43.140 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:43.140 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:43.141 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:43.141 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:43.141 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:43.149 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.148+0000 7fc0c6ffd640 1 -- 172.21.15.156:0/3578355352 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1105589837 0 0) 0x7fc0d8136c00 con 0x7fc0d81430c0 2024-03-25T18:12:43.149 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.148+0000 7fc0c6ffd640 1 -- 172.21.15.156:0/3578355352 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0d813b600 con 0x7fc0d81430c0 2024-03-25T18:12:43.149 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.148+0000 7fc0c6ffd640 1 -- 172.21.15.156:0/3578355352 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc0d0001bd0 con 0x7fc0d81430c0 2024-03-25T18:12:43.149 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.148+0000 7fc0c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:43.149 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.148+0000 7fc0c6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:43.149 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.148+0000 7fc0c6ffd640 10 monclient(hunting): dump: 2024-03-25T18:12:43.149 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:43.149 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:43.149 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:43.149 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:43.149 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:43.150 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:43.150 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:43.150 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:43.150 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:43.150 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:43.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.151+0000 7fc0c6ffd640 1 -- 172.21.15.156:0/3578355352 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 737485235 0 0) 0x7fc0d813b260 con 0x7fc0d813f710 2024-03-25T18:12:43.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.151+0000 7fc0c6ffd640 1 -- 172.21.15.156:0/3578355352 >> v1:172.21.15.156:6790/0 conn(0x7fc0d81430c0 legacy=0x7fc0d81e1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:43.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.151+0000 7fc0c6ffd640 1 -- 172.21.15.156:0/3578355352 >> v1:172.21.15.156:6789/0 conn(0x7fc0d80a98f0 legacy=0x7fc0d81da990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:43.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.151+0000 7fc0c6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:12:43.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.151+0000 7fc0c6ffd640 1 monclient: found mon.b 2024-03-25T18:12:43.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.151+0000 7fc0c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:43.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.151+0000 7fc0c6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:12:43.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.151+0000 7fc0c6ffd640 1 -- 172.21.15.156:0/3578355352 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d81352b0 con 0x7fc0d813f710 2024-03-25T18:12:43.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.151+0000 7fc0c6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:12:43.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.151+0000 7fc0c6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:12:43.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.151+0000 7fc0c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:43.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.151+0000 7fc0dce74640 5 monclient: authenticate success, global_id 4423 2024-03-25T18:12:43.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.151+0000 7fc0dce74640 10 monclient: _renew_subs 2024-03-25T18:12:43.154 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.151+0000 7fc0dce74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:12:43.154 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.151+0000 7fc0dce74640 1 -- 172.21.15.156:0/3578355352 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d8135500 con 0x7fc0d813f710 2024-03-25T18:12:43.154 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.151+0000 7fc0dce74640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:43.154 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.151+0000 7fc0dce74640 10 monclient: _renew_subs 2024-03-25T18:12:43.154 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.151+0000 7fc0dce74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:12:43.154 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.151+0000 7fc0dce74640 1 -- 172.21.15.156:0/3578355352 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d80b7ae0 con 0x7fc0d813f710 2024-03-25T18:12:43.154 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.151+0000 7fc0dce74640 1 librados: init done 2024-03-25T18:12:43.155 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.152+0000 7fc0dce74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:43.155 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.152+0000 7fc0dce74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:43.155 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.152+0000 7fc0dce74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:12:43.155 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.152+0000 7fc0dce74640 1 -- 172.21.15.156:0/3578355352 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d813b790 con 0x7fc0d813f710 2024-03-25T18:12:43.437 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.436+0000 7fc0c6ffd640 1 -- 172.21.15.156:0/3578355352 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0cc004a90 con 0x7fc0d813f710 2024-03-25T18:12:43.437 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.436+0000 7fc0c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:43.437 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.436+0000 7fc0c6ffd640 1 -- 172.21.15.156:0/3578355352 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc0cc004e20 con 0x7fc0d813f710 2024-03-25T18:12:43.437 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.436+0000 7fc0c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:43.437 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.436+0000 7fc0c6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:43.437 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.436+0000 7fc0c6ffd640 10 monclient: dump: 2024-03-25T18:12:43.437 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:43.437 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:43.437 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:43.438 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:43.438 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:43.438 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:43.438 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:43.438 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:43.438 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:43.438 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:43.438 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.436+0000 7fc0d4d99640 4 set_mon_vals no callback set 2024-03-25T18:12:43.439 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.437+0000 7fc0c6ffd640 1 -- 172.21.15.156:0/3578355352 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7fc0cc005040 con 0x7fc0d813f710 2024-03-25T18:12:43.439 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.437+0000 7fc0d659c640 10 monclient: get_auth_request con 0x7fc0b806e0f0 auth_method 0 2024-03-25T18:12:43.439 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.438+0000 7fc0c6ffd640 10 client.4423.objecter ms_handle_connect 0x7fc0b806e0f0 2024-03-25T18:12:43.442 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.441+0000 7fc0c6ffd640 1 -- 172.21.15.156:0/3578355352 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7fc0cc054920 con 0x7fc0d813f710 2024-03-25T18:12:43.442 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.441+0000 7fc0c6ffd640 10 client.4423.objecter ms_dispatch 0x7fc0d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:43.442 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.441+0000 7fc0c6ffd640 3 client.4423.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:43.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.441+0000 7fc0c6ffd640 3 client.4423.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:43.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.441+0000 7fc0c6ffd640 20 client.4423.objecter dump_active .. 0 homeless 2024-03-25T18:12:43.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.441+0000 7fc0c6ffd640 1 -- 172.21.15.156:0/3578355352 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc0cc050260 con 0x7fc0d813f710 2024-03-25T18:12:43.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.441+0000 7fc0c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:43.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.441+0000 7fc0c6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:43.687 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.686+0000 7fc0dce74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:43.687 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.686+0000 7fc0dce74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:43.687 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.686+0000 7fc0dce74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:12:43.687 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.686+0000 7fc0dce74640 1 -- 172.21.15.156:0/3578355352 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0d813b3f0 con 0x7fc0d813f710 2024-03-25T18:12:43.742 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.741+0000 7fc0c6ffd640 1 -- 172.21.15.156:0/3578355352 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7fc0cc052360 con 0x7fc0d813f710 2024-03-25T18:12:43.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.741+0000 7fc0c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:43.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.741+0000 7fc0c6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:43.743 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:12:43.743 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:43.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.744+0000 7fc0dce74640 10 librados: watch_flush enter 2024-03-25T18:12:43.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.744+0000 7fc0dce74640 10 librados: watch_flush exit 2024-03-25T18:12:43.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.744+0000 7fc0dce74640 20 client.4423.objecter shutdown clearing up homeless session... 2024-03-25T18:12:43.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.744+0000 7fc0dce74640 10 client.4423.objecter successfully canceled tick 2024-03-25T18:12:43.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.744+0000 7fc0dce74640 1 -- 172.21.15.156:0/3578355352 >> v1:172.21.15.194:6800/2946427713 conn(0x7fc0b806e0f0 legacy=0x7fc0b80705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:43.746 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.744+0000 7fc0dce74640 10 monclient: shutdown 2024-03-25T18:12:43.746 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.744+0000 7fc0dce74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:43.746 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.744+0000 7fc0dce74640 1 -- 172.21.15.156:0/3578355352 >> v1:172.21.15.194:6789/0 conn(0x7fc0d813f710 legacy=0x7fc0d81db0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:43.746 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.744+0000 7fc0dce74640 1 -- 172.21.15.156:0/3578355352 shutdown_connections 2024-03-25T18:12:43.746 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.744+0000 7fc0dce74640 1 -- 172.21.15.156:0/3578355352 >> 172.21.15.156:0/3578355352 conn(0x7fc0d80b7ea0 msgr2=0x7fc0d813d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:43.746 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.744+0000 7fc0dce74640 1 -- 172.21.15.156:0/3578355352 shutdown_connections 2024-03-25T18:12:43.746 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.744+0000 7fc0dce74640 1 -- 172.21.15.156:0/3578355352 wait complete. 2024-03-25T18:12:43.746 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:43.744+0000 7fc0dce74640 1 librados: shutdown 2024-03-25T18:12:43.758 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:44.048 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:04:50.038 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:44.060 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:04:50.050 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:44.093 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:04:50.084 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:44.138 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:04:50.105 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:44.145 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:04:50.112 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:44.154 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:04:50.144 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:44.163 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:04:50.154 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:44.180 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:04:50.147 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:44.199 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:04:50.190 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:50.565 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:50.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.676+0000 7f87d41a3640 10 monclient: get_monmap_and_config 2024-03-25T18:12:50.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.676+0000 7f87d41a3640 10 monclient: build_initial_monmap 2024-03-25T18:12:50.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.676+0000 7f87d41a3640 10 monclient: monmap: 2024-03-25T18:12:50.678 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:12:50.678 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:50.678 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:12:50.677719+0000 2024-03-25T18:12:50.678 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:12:50.677719+0000 2024-03-25T18:12:50.678 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:50.678 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:50.678 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:12:50.678 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:12:50.678 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:12:50.678 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:50.679 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.677+0000 7f87d41a3640 1 Processor -- start 2024-03-25T18:12:50.679 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.677+0000 7f87d41a3640 1 -- start start 2024-03-25T18:12:50.680 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.677+0000 7f87d41a3640 10 monclient: init 2024-03-25T18:12:50.680 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.678+0000 7f87d41a3640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:50.680 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.678+0000 7f87d41a3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:12:50.680 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.678+0000 7f87d41a3640 10 monclient(hunting): picked mon.noname-a con 0x7f87cc0a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:12:50.680 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.678+0000 7f87d41a3640 10 monclient(hunting): picked mon.noname-b con 0x7f87cc13f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:12:50.680 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.678+0000 7f87d41a3640 10 monclient(hunting): picked mon.noname-c con 0x7f87cc1430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:12:50.680 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.678+0000 7f87d41a3640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87cc142520 con 0x7f87cc0a98f0 2024-03-25T18:12:50.680 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.678+0000 7f87d41a3640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87cc0b28a0 con 0x7f87cc1430c0 2024-03-25T18:12:50.680 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.678+0000 7f87d41a3640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87cc0b2a30 con 0x7f87cc13f710 2024-03-25T18:12:50.680 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.678+0000 7f87d41a3640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:50.680 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.678+0000 7f87d41a3640 10 monclient(hunting): authenticate will time out at 7251.989225s 2024-03-25T18:12:50.687 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.686+0000 7f87d1f3f640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f87cc0a98f0 0x7f87cc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:53852/0 (socket says 172.21.15.156:53852) 2024-03-25T18:12:50.687 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.686+0000 7f87d1f3f640 1 -- 172.21.15.156:0/3663965777 learned_addr learned my addr 172.21.15.156:0/3663965777 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:12:50.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.696+0000 7f87d0f3d640 1 -- 172.21.15.156:0/3663965777 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1313276544 0 0) 0x7f87cc142520 con 0x7f87cc0a98f0 2024-03-25T18:12:50.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.696+0000 7f87d0f3d640 10 monclient(hunting): _negotiate 2024-03-25T18:12:50.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.696+0000 7f87d0f3d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:50.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.696+0000 7f87d0f3d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:50.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.696+0000 7f87d0f3d640 10 monclient(hunting): my global_id is 4425 2024-03-25T18:12:50.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.696+0000 7f87d0f3d640 1 -- 172.21.15.156:0/3663965777 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87b8003520 con 0x7f87cc0a98f0 2024-03-25T18:12:50.707 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.706+0000 7f87d0f3d640 1 -- 172.21.15.156:0/3663965777 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403092502 0 0) 0x7f87cc0b2a30 con 0x7f87cc13f710 2024-03-25T18:12:50.707 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.706+0000 7f87d0f3d640 10 monclient(hunting): _negotiate 2024-03-25T18:12:50.707 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.706+0000 7f87d0f3d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:50.707 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.706+0000 7f87d0f3d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:50.707 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.706+0000 7f87d0f3d640 10 monclient(hunting): my global_id is 4433 2024-03-25T18:12:50.707 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.706+0000 7f87d0f3d640 1 -- 172.21.15.156:0/3663965777 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87cc142520 con 0x7f87cc13f710 2024-03-25T18:12:50.712 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.711+0000 7f87d0f3d640 1 -- 172.21.15.156:0/3663965777 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3938887270 0 0) 0x7f87cc0b28a0 con 0x7f87cc1430c0 2024-03-25T18:12:50.712 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.711+0000 7f87d0f3d640 10 monclient(hunting): _negotiate 2024-03-25T18:12:50.712 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.711+0000 7f87d0f3d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:50.712 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.711+0000 7f87d0f3d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:50.712 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.711+0000 7f87d0f3d640 10 monclient(hunting): my global_id is 4426 2024-03-25T18:12:50.712 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.711+0000 7f87d0f3d640 1 -- 172.21.15.156:0/3663965777 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87cc0b2a30 con 0x7f87cc1430c0 2024-03-25T18:12:50.725 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.723+0000 7f87d0f3d640 1 -- 172.21.15.156:0/3663965777 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 398698960 0 0) 0x7f87b8003520 con 0x7f87cc0a98f0 2024-03-25T18:12:50.725 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.723+0000 7f87d0f3d640 1 -- 172.21.15.156:0/3663965777 >> v1:172.21.15.156:6790/0 conn(0x7f87cc13f710 legacy=0x7f87cc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:50.725 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.723+0000 7f87d0f3d640 1 -- 172.21.15.156:0/3663965777 >> v1:172.21.15.194:6789/0 conn(0x7f87cc1430c0 legacy=0x7f87cc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:50.725 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.723+0000 7f87d0f3d640 10 monclient: _finish_hunting 0 2024-03-25T18:12:50.725 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.723+0000 7f87d0f3d640 1 monclient: found mon.noname-a 2024-03-25T18:12:50.725 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.723+0000 7f87d0f3d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:50.725 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.723+0000 7f87d0f3d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:12:50.725 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.723+0000 7f87d0f3d640 1 -- 172.21.15.156:0/3663965777 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87cc0b44c0 con 0x7f87cc0a98f0 2024-03-25T18:12:50.725 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.723+0000 7f87d0f3d640 10 monclient: _finish_auth 0 2024-03-25T18:12:50.725 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.723+0000 7f87d0f3d640 10 monclient: _check_auth_tickets 2024-03-25T18:12:50.725 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.723+0000 7f87d0f3d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:50.725 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.723+0000 7f87d0f3d640 1 -- 172.21.15.156:0/3663965777 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f87c8001a10 con 0x7f87cc0a98f0 2024-03-25T18:12:50.726 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.723+0000 7f87d0f3d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:50.726 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.723+0000 7f87d0f3d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:50.726 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.723+0000 7f87d0f3d640 10 monclient: dump: 2024-03-25T18:12:50.726 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:50.726 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:50.726 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:50.726 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:50.726 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:50.726 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:50.726 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:50.726 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:50.726 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:50.726 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:50.727 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.723+0000 7f87d41a3640 5 monclient: authenticate success, global_id 4425 2024-03-25T18:12:50.727 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.723+0000 7f87d41a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:50.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.725+0000 7f87d0f3d640 1 -- 172.21.15.156:0/3663965777 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87c8001b70 con 0x7f87cc0a98f0 2024-03-25T18:12:50.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.725+0000 7f87d0f3d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:50.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.725+0000 7f87d0f3d640 1 -- 172.21.15.156:0/3663965777 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f87c8001f80 con 0x7f87cc0a98f0 2024-03-25T18:12:50.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.725+0000 7f87d41a3640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:50.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.725+0000 7f87d41a3640 4 set_mon_vals no callback set 2024-03-25T18:12:50.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.725+0000 7f87d0f3d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:50.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.725+0000 7f87d0f3d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:50.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.725+0000 7f87d0f3d640 10 monclient: dump: 2024-03-25T18:12:50.728 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:50.728 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:50.728 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:50.729 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:50.729 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:50.729 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:50.729 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:50.729 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:50.729 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:50.729 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:50.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.726+0000 7f87d41a3640 10 monclient: shutdown 2024-03-25T18:12:50.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.726+0000 7f87d41a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:50.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.726+0000 7f87d41a3640 1 -- 172.21.15.156:0/3663965777 >> v1:172.21.15.156:6789/0 conn(0x7f87cc0a98f0 legacy=0x7f87cc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:50.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.726+0000 7f87d41a3640 1 -- 172.21.15.156:0/3663965777 shutdown_connections 2024-03-25T18:12:50.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.726+0000 7f87d41a3640 1 -- 172.21.15.156:0/3663965777 >> 172.21.15.156:0/3663965777 conn(0x7f87cc0b7ea0 msgr2=0x7f87cc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:50.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.726+0000 7f87d41a3640 1 -- 172.21.15.156:0/3663965777 shutdown_connections 2024-03-25T18:12:50.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.726+0000 7f87d41a3640 1 -- 172.21.15.156:0/3663965777 wait complete. 2024-03-25T18:12:50.731 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.727+0000 7f87d41a3640 10 monclient: build_initial_monmap 2024-03-25T18:12:50.731 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.727+0000 7f87d41a3640 10 monclient: monmap: 2024-03-25T18:12:50.731 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:12:50.731 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:50.731 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:12:50.731 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:12:50.731 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:50.731 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:12:50.732 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:12:50.732 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:12:50.732 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:12:50.732 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:50.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.728+0000 7f87d41a3640 1 librados: starting msgr at 2024-03-25T18:12:50.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.728+0000 7f87d41a3640 1 librados: starting objecter 2024-03-25T18:12:50.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.728+0000 7f87d41a3640 1 Processor -- start 2024-03-25T18:12:50.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.728+0000 7f87d41a3640 1 -- start start 2024-03-25T18:12:50.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.728+0000 7f87d41a3640 1 librados: setting wanted keys 2024-03-25T18:12:50.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.728+0000 7f87d41a3640 1 librados: calling monclient init 2024-03-25T18:12:50.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.728+0000 7f87d41a3640 10 monclient: init 2024-03-25T18:12:50.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.728+0000 7f87d41a3640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:50.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.728+0000 7f87d41a3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:12:50.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.728+0000 7f87d41a3640 10 monclient(hunting): picked mon.noname-b con 0x7f87cc0a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:12:50.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.729+0000 7f87d41a3640 10 monclient(hunting): picked mon.noname-a con 0x7f87cc13f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:12:50.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.729+0000 7f87d41a3640 10 monclient(hunting): picked mon.noname-c con 0x7f87cc1430c0 addr v1:172.21.15.156:6790/0 2024-03-25T18:12:50.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.729+0000 7f87d41a3640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87cc1369b0 con 0x7f87cc13f710 2024-03-25T18:12:50.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.729+0000 7f87d41a3640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87cc1366b0 con 0x7f87cc0a98f0 2024-03-25T18:12:50.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.729+0000 7f87d41a3640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87cc133ad0 con 0x7f87cc1430c0 2024-03-25T18:12:50.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.729+0000 7f87d41a3640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:50.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.729+0000 7f87d41a3640 10 monclient(hunting): authenticate will time out at 7252.039884s 2024-03-25T18:12:50.739 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.738+0000 7f87d173e640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f87cc13f710 0x7f87cc132cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:53912/0 (socket says 172.21.15.156:53912) 2024-03-25T18:12:50.739 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.738+0000 7f87d173e640 1 -- 172.21.15.156:0/231027222 learned_addr learned my addr 172.21.15.156:0/231027222 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:12:50.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.744+0000 7f87c27fc640 10 client.?.objecter ms_handle_connect 0x7f87cc13f710 2024-03-25T18:12:50.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.744+0000 7f87c27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:50.748 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.747+0000 7f87c27fc640 10 client.?.objecter ms_handle_connect 0x7f87cc0a98f0 2024-03-25T18:12:50.748 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.747+0000 7f87c27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:50.754 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.753+0000 7f87c27fc640 10 client.?.objecter ms_handle_connect 0x7f87cc1430c0 2024-03-25T18:12:50.754 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.753+0000 7f87c27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:50.757 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.757+0000 7f87c27fc640 1 -- 172.21.15.156:0/231027222 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 960364205 0 0) 0x7f87cc1366b0 con 0x7f87cc0a98f0 2024-03-25T18:12:50.758 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.757+0000 7f87c27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:50.758 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.757+0000 7f87c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:50.758 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.757+0000 7f87c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:50.758 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.757+0000 7f87c27fc640 10 monclient(hunting): my global_id is 4429 2024-03-25T18:12:50.758 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.757+0000 7f87c27fc640 1 -- 172.21.15.156:0/231027222 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87b0004eb0 con 0x7f87cc0a98f0 2024-03-25T18:12:50.760 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.759+0000 7f87c27fc640 1 -- 172.21.15.156:0/231027222 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2213639457 0 0) 0x7f87cc1369b0 con 0x7f87cc13f710 2024-03-25T18:12:50.760 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.759+0000 7f87c27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:50.760 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.759+0000 7f87c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:50.760 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.759+0000 7f87c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:50.760 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.759+0000 7f87c27fc640 10 monclient(hunting): my global_id is 4428 2024-03-25T18:12:50.760 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.759+0000 7f87c27fc640 1 -- 172.21.15.156:0/231027222 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87cc1366b0 con 0x7f87cc13f710 2024-03-25T18:12:50.774 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.773+0000 7f87c27fc640 1 -- 172.21.15.156:0/231027222 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1499491978 0 0) 0x7f87cc133ad0 con 0x7f87cc1430c0 2024-03-25T18:12:50.774 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.773+0000 7f87c27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:50.774 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.773+0000 7f87c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:50.774 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.773+0000 7f87c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:50.774 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.773+0000 7f87c27fc640 10 monclient(hunting): my global_id is 4436 2024-03-25T18:12:50.774 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.773+0000 7f87c27fc640 1 -- 172.21.15.156:0/231027222 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87cc1369b0 con 0x7f87cc1430c0 2024-03-25T18:12:50.775 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.773+0000 7f87c27fc640 1 -- 172.21.15.156:0/231027222 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 732808958 0 0) 0x7f87cc1366b0 con 0x7f87cc13f710 2024-03-25T18:12:50.775 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.774+0000 7f87c27fc640 1 -- 172.21.15.156:0/231027222 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87cc133ad0 con 0x7f87cc13f710 2024-03-25T18:12:50.775 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.774+0000 7f87c27fc640 1 -- 172.21.15.156:0/231027222 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f87bc0019b0 con 0x7f87cc13f710 2024-03-25T18:12:50.775 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.774+0000 7f87c27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:50.775 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.774+0000 7f87c27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:50.775 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.774+0000 7f87c27fc640 10 monclient(hunting): dump: 2024-03-25T18:12:50.776 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:50.776 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:50.776 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:50.776 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:50.776 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:50.776 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:50.776 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:50.776 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:50.776 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:50.776 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:50.776 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.774+0000 7f87c27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:50.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.776+0000 7f87c27fc640 1 -- 172.21.15.156:0/231027222 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2920263202 0 0) 0x7f87b0004eb0 con 0x7f87cc0a98f0 2024-03-25T18:12:50.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.776+0000 7f87c27fc640 1 -- 172.21.15.156:0/231027222 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87cc1366b0 con 0x7f87cc0a98f0 2024-03-25T18:12:50.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.776+0000 7f87c27fc640 1 -- 172.21.15.156:0/231027222 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f87c8001e80 con 0x7f87cc0a98f0 2024-03-25T18:12:50.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.776+0000 7f87c27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:50.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.776+0000 7f87c27fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:50.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.776+0000 7f87c27fc640 10 monclient(hunting): dump: 2024-03-25T18:12:50.777 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:50.777 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:50.777 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:50.777 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:50.777 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:50.778 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:50.778 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:50.778 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:50.778 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:50.778 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:50.785 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.783+0000 7f87c27fc640 1 -- 172.21.15.156:0/231027222 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 433981265 0 0) 0x7f87cc133ad0 con 0x7f87cc13f710 2024-03-25T18:12:50.785 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.783+0000 7f87c27fc640 1 -- 172.21.15.156:0/231027222 >> v1:172.21.15.156:6790/0 conn(0x7f87cc1430c0 legacy=0x7f87cc1333c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:50.785 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.783+0000 7f87c27fc640 1 -- 172.21.15.156:0/231027222 >> v1:172.21.15.194:6789/0 conn(0x7f87cc0a98f0 legacy=0x7f87cc1371e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:50.785 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.783+0000 7f87c27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:12:50.785 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.783+0000 7f87c27fc640 1 monclient: found mon.a 2024-03-25T18:12:50.785 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.783+0000 7f87c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:50.785 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.783+0000 7f87c27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:12:50.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.783+0000 7f87c27fc640 1 -- 172.21.15.156:0/231027222 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87cc136e40 con 0x7f87cc13f710 2024-03-25T18:12:50.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.783+0000 7f87c27fc640 10 monclient: _finish_auth 0 2024-03-25T18:12:50.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.783+0000 7f87c27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:12:50.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.783+0000 7f87c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:50.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.783+0000 7f87d41a3640 5 monclient: authenticate success, global_id 4428 2024-03-25T18:12:50.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.783+0000 7f87d41a3640 10 monclient: _renew_subs 2024-03-25T18:12:50.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.783+0000 7f87d41a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:12:50.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.783+0000 7f87d41a3640 1 -- 172.21.15.156:0/231027222 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87cc1378f0 con 0x7f87cc13f710 2024-03-25T18:12:50.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.783+0000 7f87d41a3640 10 client.4428.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:50.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.783+0000 7f87d41a3640 10 monclient: _renew_subs 2024-03-25T18:12:50.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.783+0000 7f87d41a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:12:50.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.783+0000 7f87d41a3640 1 -- 172.21.15.156:0/231027222 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87cc0b1480 con 0x7f87cc13f710 2024-03-25T18:12:50.787 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.783+0000 7f87d41a3640 1 librados: init done 2024-03-25T18:12:50.787 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.784+0000 7f87d41a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:50.787 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.784+0000 7f87d41a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:50.787 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.784+0000 7f87d41a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:12:50.787 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:50.784+0000 7f87d41a3640 1 -- 172.21.15.156:0/231027222 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87cc137030 con 0x7f87cc13f710 2024-03-25T18:12:51.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.074+0000 7f87c27fc640 1 -- 172.21.15.156:0/231027222 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87bc003610 con 0x7f87cc13f710 2024-03-25T18:12:51.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.074+0000 7f87c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:51.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.074+0000 7f87c27fc640 1 -- 172.21.15.156:0/231027222 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f87bc0039e0 con 0x7f87cc13f710 2024-03-25T18:12:51.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.074+0000 7f87c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:51.077 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.074+0000 7f87c27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:51.077 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.074+0000 7f87c27fc640 10 monclient: dump: 2024-03-25T18:12:51.077 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:51.077 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:51.077 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:51.077 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:51.077 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:51.077 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:51.077 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:51.077 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:51.077 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:51.077 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:51.077 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.074+0000 7f87c3fff640 4 set_mon_vals no callback set 2024-03-25T18:12:51.078 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.075+0000 7f87c27fc640 1 -- 172.21.15.156:0/231027222 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f87bc003c00 con 0x7f87cc13f710 2024-03-25T18:12:51.078 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.076+0000 7f87d1f3f640 10 monclient: get_auth_request con 0x7f87b006df10 auth_method 0 2024-03-25T18:12:51.078 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.076+0000 7f87c27fc640 10 client.4428.objecter ms_handle_connect 0x7f87b006df10 2024-03-25T18:12:51.083 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.080+0000 7f87c27fc640 1 -- 172.21.15.156:0/231027222 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f87bc054810 con 0x7f87cc13f710 2024-03-25T18:12:51.083 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.080+0000 7f87c27fc640 10 client.4428.objecter ms_dispatch 0x7f87cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:51.083 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.080+0000 7f87c27fc640 3 client.4428.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:51.083 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.080+0000 7f87c27fc640 3 client.4428.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:51.083 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.080+0000 7f87c27fc640 20 client.4428.objecter dump_active .. 0 homeless 2024-03-25T18:12:51.083 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.080+0000 7f87c27fc640 1 -- 172.21.15.156:0/231027222 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f87bc050320 con 0x7f87cc13f710 2024-03-25T18:12:51.083 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.080+0000 7f87c27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:51.083 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.080+0000 7f87c27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:51.330 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.328+0000 7f87d41a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:51.331 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.328+0000 7f87d41a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:51.331 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.328+0000 7f87d41a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:12:51.331 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.328+0000 7f87d41a3640 1 -- 172.21.15.156:0/231027222 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87cc136b60 con 0x7f87cc13f710 2024-03-25T18:12:51.369 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.367+0000 7f87c27fc640 1 -- 172.21.15.156:0/231027222 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f87cc136b60 con 0x7f87cc13f710 2024-03-25T18:12:51.370 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.367+0000 7f87c27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:51.370 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.367+0000 7f87c27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:51.370 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:12:51.370 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:51.373 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.370+0000 7f87d41a3640 10 librados: watch_flush enter 2024-03-25T18:12:51.373 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.370+0000 7f87d41a3640 10 librados: watch_flush exit 2024-03-25T18:12:51.373 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.370+0000 7f87d41a3640 20 client.4428.objecter shutdown clearing up homeless session... 2024-03-25T18:12:51.373 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.370+0000 7f87d41a3640 10 client.4428.objecter successfully canceled tick 2024-03-25T18:12:51.373 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.370+0000 7f87d41a3640 1 -- 172.21.15.156:0/231027222 >> v1:172.21.15.194:6800/2946427713 conn(0x7f87b006df10 legacy=0x7f87b00703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:51.373 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.370+0000 7f87d41a3640 10 monclient: shutdown 2024-03-25T18:12:51.373 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.370+0000 7f87d41a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:51.373 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.370+0000 7f87d41a3640 1 -- 172.21.15.156:0/231027222 >> v1:172.21.15.156:6789/0 conn(0x7f87cc13f710 legacy=0x7f87cc132cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:51.373 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.370+0000 7f87d41a3640 1 -- 172.21.15.156:0/231027222 shutdown_connections 2024-03-25T18:12:51.373 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.370+0000 7f87d41a3640 1 -- 172.21.15.156:0/231027222 >> 172.21.15.156:0/231027222 conn(0x7f87cc0b7ea0 msgr2=0x7f87cc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:51.373 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.370+0000 7f87d41a3640 1 -- 172.21.15.156:0/231027222 shutdown_connections 2024-03-25T18:12:51.373 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.370+0000 7f87d41a3640 1 -- 172.21.15.156:0/231027222 wait complete. 2024-03-25T18:12:51.373 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:51.370+0000 7f87d41a3640 1 librados: shutdown 2024-03-25T18:12:51.384 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:58.191 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:58.294 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.293+0000 7fe46619d640 10 monclient: get_monmap_and_config 2024-03-25T18:12:58.295 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.293+0000 7fe46619d640 10 monclient: build_initial_monmap 2024-03-25T18:12:58.295 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.293+0000 7fe46619d640 10 monclient: monmap: 2024-03-25T18:12:58.295 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:12:58.295 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:58.295 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:12:58.294799+0000 2024-03-25T18:12:58.295 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:12:58.294799+0000 2024-03-25T18:12:58.295 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:58.295 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:58.295 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:12:58.295 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:12:58.295 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:12:58.295 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:58.297 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.294+0000 7fe46619d640 1 Processor -- start 2024-03-25T18:12:58.297 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.294+0000 7fe46619d640 1 -- start start 2024-03-25T18:12:58.297 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.294+0000 7fe46619d640 10 monclient: init 2024-03-25T18:12:58.297 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.295+0000 7fe46619d640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:58.297 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.295+0000 7fe46619d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:12:58.297 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.295+0000 7fe46619d640 10 monclient(hunting): picked mon.noname-b con 0x7fe4600b4d60 addr v1:172.21.15.156:6790/0 2024-03-25T18:12:58.297 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.295+0000 7fe46619d640 10 monclient(hunting): picked mon.noname-c con 0x7fe4600b1730 addr v1:172.21.15.194:6789/0 2024-03-25T18:12:58.297 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.295+0000 7fe46619d640 10 monclient(hunting): picked mon.noname-a con 0x7fe46014be90 addr v1:172.21.15.156:6789/0 2024-03-25T18:12:58.297 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.295+0000 7fe46619d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4600af940 con 0x7fe46014be90 2024-03-25T18:12:58.297 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.295+0000 7fe46619d640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4600b0a70 con 0x7fe4600b1730 2024-03-25T18:12:58.297 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.295+0000 7fe46619d640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4600b0d50 con 0x7fe4600b4d60 2024-03-25T18:12:58.297 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.295+0000 7fe46619d640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:58.298 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.295+0000 7fe46619d640 10 monclient(hunting): authenticate will time out at 7259.605962s 2024-03-25T18:12:58.298 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.295+0000 7fe45f7fe640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7fe4600b4d60 0x7fe46013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:42762/0 (socket says 172.21.15.156:42762) 2024-03-25T18:12:58.298 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.295+0000 7fe45f7fe640 1 -- 172.21.15.156:0/1828700862 learned_addr learned my addr 172.21.15.156:0/1828700862 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:12:58.323 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.322+0000 7fe45e7fc640 1 -- 172.21.15.156:0/1828700862 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3174405059 0 0) 0x7fe4600b0a70 con 0x7fe4600b1730 2024-03-25T18:12:58.323 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.322+0000 7fe45e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:58.323 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.322+0000 7fe45e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:58.323 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.322+0000 7fe45e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:58.323 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.322+0000 7fe45e7fc640 10 monclient(hunting): my global_id is 4432 2024-03-25T18:12:58.323 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.322+0000 7fe45e7fc640 1 -- 172.21.15.156:0/1828700862 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe44c003520 con 0x7fe4600b1730 2024-03-25T18:12:58.338 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.337+0000 7fe45e7fc640 1 -- 172.21.15.156:0/1828700862 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 889650098 0 0) 0x7fe4600af940 con 0x7fe46014be90 2024-03-25T18:12:58.338 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.337+0000 7fe45e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:58.338 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.337+0000 7fe45e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:58.338 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.337+0000 7fe45e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:58.339 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.337+0000 7fe45e7fc640 10 monclient(hunting): my global_id is 4431 2024-03-25T18:12:58.339 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.337+0000 7fe45e7fc640 1 -- 172.21.15.156:0/1828700862 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4600b0a70 con 0x7fe46014be90 2024-03-25T18:12:58.350 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.349+0000 7fe45e7fc640 1 -- 172.21.15.156:0/1828700862 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3444337767 0 0) 0x7fe4600b0a70 con 0x7fe46014be90 2024-03-25T18:12:58.350 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.349+0000 7fe45e7fc640 1 -- 172.21.15.156:0/1828700862 >> v1:172.21.15.156:6790/0 conn(0x7fe4600b4d60 legacy=0x7fe46013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:58.350 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.349+0000 7fe45e7fc640 1 -- 172.21.15.156:0/1828700862 >> v1:172.21.15.194:6789/0 conn(0x7fe4600b1730 legacy=0x7fe4600b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:58.350 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.349+0000 7fe45e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:12:58.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.349+0000 7fe45e7fc640 1 monclient: found mon.noname-a 2024-03-25T18:12:58.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.349+0000 7fe45e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:58.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.349+0000 7fe45e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:12:58.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.349+0000 7fe45e7fc640 1 -- 172.21.15.156:0/1828700862 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4600b1030 con 0x7fe46014be90 2024-03-25T18:12:58.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.349+0000 7fe45e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:12:58.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.349+0000 7fe45e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:12:58.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.349+0000 7fe45e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:58.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.349+0000 7fe45e7fc640 1 -- 172.21.15.156:0/1828700862 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fe458001670 con 0x7fe46014be90 2024-03-25T18:12:58.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.349+0000 7fe46619d640 5 monclient: authenticate success, global_id 4431 2024-03-25T18:12:58.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.349+0000 7fe46619d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:58.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.349+0000 7fe45e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:58.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.349+0000 7fe45e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:58.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.349+0000 7fe45e7fc640 10 monclient: dump: 2024-03-25T18:12:58.351 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:58.352 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:58.352 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:58.352 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:58.352 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:58.352 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:58.352 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:58.352 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:58.352 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:58.352 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:58.352 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.349+0000 7fe46619d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:58.364 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.363+0000 7fe45e7fc640 1 -- 172.21.15.156:0/1828700862 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4580017d0 con 0x7fe46014be90 2024-03-25T18:12:58.365 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.363+0000 7fe45e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:58.365 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.363+0000 7fe45e7fc640 1 -- 172.21.15.156:0/1828700862 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fe458001ba0 con 0x7fe46014be90 2024-03-25T18:12:58.365 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.363+0000 7fe45e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:58.365 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.363+0000 7fe45e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:58.365 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.363+0000 7fe45e7fc640 10 monclient: dump: 2024-03-25T18:12:58.365 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:58.365 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:58.365 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:58.365 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:58.365 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:58.365 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:58.365 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:58.365 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:58.365 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:58.365 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:58.366 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.363+0000 7fe46619d640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:58.366 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.363+0000 7fe46619d640 4 set_mon_vals no callback set 2024-03-25T18:12:58.368 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.364+0000 7fe46619d640 10 monclient: shutdown 2024-03-25T18:12:58.368 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.364+0000 7fe46619d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:58.368 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.364+0000 7fe46619d640 1 -- 172.21.15.156:0/1828700862 >> v1:172.21.15.156:6789/0 conn(0x7fe46014be90 legacy=0x7fe46014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:58.368 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.364+0000 7fe46619d640 1 -- 172.21.15.156:0/1828700862 shutdown_connections 2024-03-25T18:12:58.369 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.364+0000 7fe46619d640 1 -- 172.21.15.156:0/1828700862 >> 172.21.15.156:0/1828700862 conn(0x7fe4600617a0 msgr2=0x7fe460061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:58.369 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.364+0000 7fe46619d640 1 -- 172.21.15.156:0/1828700862 shutdown_connections 2024-03-25T18:12:58.369 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.364+0000 7fe46619d640 1 -- 172.21.15.156:0/1828700862 wait complete. 2024-03-25T18:12:58.369 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.365+0000 7fe46619d640 10 monclient: build_initial_monmap 2024-03-25T18:12:58.369 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.365+0000 7fe46619d640 10 monclient: monmap: 2024-03-25T18:12:58.369 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:12:58.369 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:58.369 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:12:58.369 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:12:58.369 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:58.369 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:12:58.369 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:12:58.369 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:12:58.369 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:12:58.370 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:58.370 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.365+0000 7fe46619d640 1 librados: starting msgr at 2024-03-25T18:12:58.370 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.365+0000 7fe46619d640 1 librados: starting objecter 2024-03-25T18:12:58.370 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.365+0000 7fe46619d640 1 Processor -- start 2024-03-25T18:12:58.370 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.366+0000 7fe46619d640 1 -- start start 2024-03-25T18:12:58.370 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.366+0000 7fe46619d640 1 librados: setting wanted keys 2024-03-25T18:12:58.370 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.366+0000 7fe46619d640 1 librados: calling monclient init 2024-03-25T18:12:58.370 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.366+0000 7fe46619d640 10 monclient: init 2024-03-25T18:12:58.370 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.366+0000 7fe46619d640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:58.370 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.366+0000 7fe46619d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:12:58.370 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.366+0000 7fe46619d640 10 monclient(hunting): picked mon.noname-a con 0x7fe4600b1730 addr v1:172.21.15.156:6789/0 2024-03-25T18:12:58.370 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.366+0000 7fe46619d640 10 monclient(hunting): picked mon.noname-b con 0x7fe4600b4d60 addr v1:172.21.15.194:6789/0 2024-03-25T18:12:58.370 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.366+0000 7fe46619d640 10 monclient(hunting): picked mon.noname-c con 0x7fe46014be90 addr v1:172.21.15.156:6790/0 2024-03-25T18:12:58.371 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.366+0000 7fe46619d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe460141dc0 con 0x7fe4600b1730 2024-03-25T18:12:58.371 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.366+0000 7fe46619d640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe460140380 con 0x7fe4600b4d60 2024-03-25T18:12:58.371 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.366+0000 7fe46619d640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe460140660 con 0x7fe46014be90 2024-03-25T18:12:58.371 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.366+0000 7fe46619d640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:58.371 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.366+0000 7fe46619d640 10 monclient(hunting): authenticate will time out at 7259.677751s 2024-03-25T18:12:58.371 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.367+0000 7fe45effd640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fe4600b4d60 0x7fe4601d7690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:45770/0 (socket says 172.21.15.156:45770) 2024-03-25T18:12:58.371 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.367+0000 7fe45effd640 1 -- 172.21.15.156:0/3164943734 learned_addr learned my addr 172.21.15.156:0/3164943734 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:12:58.371 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.367+0000 7fe457fff640 10 client.?.objecter ms_handle_connect 0x7fe4600b4d60 2024-03-25T18:12:58.371 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.367+0000 7fe457fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:58.388 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.387+0000 7fe457fff640 10 client.?.objecter ms_handle_connect 0x7fe4600b1730 2024-03-25T18:12:58.388 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.387+0000 7fe457fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:58.393 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.392+0000 7fe457fff640 10 client.?.objecter ms_handle_connect 0x7fe46014be90 2024-03-25T18:12:58.393 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.392+0000 7fe457fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:58.450 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.448+0000 7fe457fff640 1 -- 172.21.15.156:0/3164943734 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372306052 0 0) 0x7fe460140380 con 0x7fe4600b4d60 2024-03-25T18:12:58.450 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.448+0000 7fe457fff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:58.450 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.448+0000 7fe457fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:58.450 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.448+0000 7fe457fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:58.450 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.448+0000 7fe457fff640 10 monclient(hunting): my global_id is 4435 2024-03-25T18:12:58.450 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.448+0000 7fe457fff640 1 -- 172.21.15.156:0/3164943734 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe440004eb0 con 0x7fe4600b4d60 2024-03-25T18:12:58.453 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.452+0000 7fe457fff640 1 -- 172.21.15.156:0/3164943734 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3089265345 0 0) 0x7fe460141dc0 con 0x7fe4600b1730 2024-03-25T18:12:58.453 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.452+0000 7fe457fff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:58.453 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.452+0000 7fe457fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:58.453 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.452+0000 7fe457fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:58.453 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.452+0000 7fe457fff640 10 monclient(hunting): my global_id is 4434 2024-03-25T18:12:58.453 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.452+0000 7fe457fff640 1 -- 172.21.15.156:0/3164943734 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe460140380 con 0x7fe4600b1730 2024-03-25T18:12:58.472 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.471+0000 7fe457fff640 1 -- 172.21.15.156:0/3164943734 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4277010187 0 0) 0x7fe460140380 con 0x7fe4600b1730 2024-03-25T18:12:58.472 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.471+0000 7fe457fff640 1 -- 172.21.15.156:0/3164943734 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe460141dc0 con 0x7fe4600b1730 2024-03-25T18:12:58.472 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.471+0000 7fe457fff640 1 -- 172.21.15.156:0/3164943734 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fe450001c20 con 0x7fe4600b1730 2024-03-25T18:12:58.472 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.471+0000 7fe457fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:58.472 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.471+0000 7fe457fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:58.472 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.471+0000 7fe457fff640 10 monclient(hunting): dump: 2024-03-25T18:12:58.472 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:58.472 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:58.472 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:58.472 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:58.472 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:58.472 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:58.472 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:58.472 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:58.473 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:58.473 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:58.473 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.471+0000 7fe457fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:58.476 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.475+0000 7fe457fff640 1 -- 172.21.15.156:0/3164943734 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2238238438 0 0) 0x7fe440004eb0 con 0x7fe4600b4d60 2024-03-25T18:12:58.476 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.475+0000 7fe457fff640 1 -- 172.21.15.156:0/3164943734 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe460140380 con 0x7fe4600b4d60 2024-03-25T18:12:58.476 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.475+0000 7fe457fff640 1 -- 172.21.15.156:0/3164943734 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fe448001a10 con 0x7fe4600b4d60 2024-03-25T18:12:58.476 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.475+0000 7fe457fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:58.476 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.475+0000 7fe457fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:58.476 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.475+0000 7fe457fff640 10 monclient(hunting): dump: 2024-03-25T18:12:58.476 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:58.476 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:58.476 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:58.476 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:58.477 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:58.477 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:58.477 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:58.477 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:58.477 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:58.477 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:58.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.485+0000 7fe457fff640 1 -- 172.21.15.156:0/3164943734 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 200648622 0 0) 0x7fe460141dc0 con 0x7fe4600b1730 2024-03-25T18:12:58.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.485+0000 7fe457fff640 1 -- 172.21.15.156:0/3164943734 >> v1:172.21.15.156:6790/0 conn(0x7fe46014be90 legacy=0x7fe4601e2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:58.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.485+0000 7fe457fff640 1 -- 172.21.15.156:0/3164943734 >> v1:172.21.15.194:6789/0 conn(0x7fe4600b4d60 legacy=0x7fe4601d7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:58.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.485+0000 7fe457fff640 10 monclient: _finish_hunting 0 2024-03-25T18:12:58.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.485+0000 7fe457fff640 1 monclient: found mon.a 2024-03-25T18:12:58.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.485+0000 7fe457fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:58.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.485+0000 7fe457fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:12:58.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.485+0000 7fe457fff640 1 -- 172.21.15.156:0/3164943734 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe460140b80 con 0x7fe4600b1730 2024-03-25T18:12:58.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.485+0000 7fe457fff640 10 monclient: _finish_auth 0 2024-03-25T18:12:58.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.485+0000 7fe457fff640 10 monclient: _check_auth_tickets 2024-03-25T18:12:58.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.485+0000 7fe457fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:58.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.485+0000 7fe46619d640 5 monclient: authenticate success, global_id 4434 2024-03-25T18:12:58.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.485+0000 7fe46619d640 10 monclient: _renew_subs 2024-03-25T18:12:58.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.485+0000 7fe46619d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:12:58.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.485+0000 7fe46619d640 1 -- 172.21.15.156:0/3164943734 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4601373e0 con 0x7fe4600b1730 2024-03-25T18:12:58.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.485+0000 7fe46619d640 10 client.4434.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:58.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.485+0000 7fe46619d640 10 monclient: _renew_subs 2024-03-25T18:12:58.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.485+0000 7fe46619d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:12:58.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.485+0000 7fe46619d640 1 -- 172.21.15.156:0/3164943734 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe460132800 con 0x7fe4600b1730 2024-03-25T18:12:58.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.485+0000 7fe46619d640 1 librados: init done 2024-03-25T18:12:58.488 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.486+0000 7fe46619d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:58.488 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.486+0000 7fe46619d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:58.488 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.486+0000 7fe46619d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:12:58.488 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.486+0000 7fe46619d640 1 -- 172.21.15.156:0/3164943734 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4600627b0 con 0x7fe4600b1730 2024-03-25T18:12:58.785 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.784+0000 7fe457fff640 1 -- 172.21.15.156:0/3164943734 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe450002030 con 0x7fe4600b1730 2024-03-25T18:12:58.785 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.784+0000 7fe457fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:58.785 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.784+0000 7fe457fff640 1 -- 172.21.15.156:0/3164943734 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fe450004d00 con 0x7fe4600b1730 2024-03-25T18:12:58.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.784+0000 7fe457fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:58.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.784+0000 7fe457fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:58.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.784+0000 7fe457fff640 10 monclient: dump: 2024-03-25T18:12:58.786 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:12:58.786 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:12:58.786 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:58.786 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:12:58.786 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:58.786 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:12:58.786 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:12:58.786 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:12:58.786 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:12:58.786 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:12:58.787 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.784+0000 7fe45dffb640 4 set_mon_vals no callback set 2024-03-25T18:12:58.787 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.784+0000 7fe457fff640 1 -- 172.21.15.156:0/3164943734 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7fe450004ee0 con 0x7fe4600b1730 2024-03-25T18:12:58.787 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.785+0000 7fe45effd640 10 monclient: get_auth_request con 0x7fe44006de90 auth_method 0 2024-03-25T18:12:58.787 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.785+0000 7fe457fff640 10 client.4434.objecter ms_handle_connect 0x7fe44006de90 2024-03-25T18:12:58.792 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.791+0000 7fe457fff640 1 -- 172.21.15.156:0/3164943734 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7fe450054a70 con 0x7fe4600b1730 2024-03-25T18:12:58.792 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.791+0000 7fe457fff640 10 client.4434.objecter ms_dispatch 0x7fe460000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:58.792 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.791+0000 7fe457fff640 3 client.4434.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:58.792 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.791+0000 7fe457fff640 3 client.4434.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:58.792 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.791+0000 7fe457fff640 20 client.4434.objecter dump_active .. 0 homeless 2024-03-25T18:12:58.792 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.791+0000 7fe457fff640 1 -- 172.21.15.156:0/3164943734 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe4500503b0 con 0x7fe4600b1730 2024-03-25T18:12:58.792 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.791+0000 7fe457fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:58.792 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:58.791+0000 7fe457fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:59.048 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:59.047+0000 7fe46619d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:59.048 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:59.047+0000 7fe46619d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:59.048 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:59.047+0000 7fe46619d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:12:59.048 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:59.047+0000 7fe46619d640 1 -- 172.21.15.156:0/3164943734 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe460069f40 con 0x7fe4600b1730 2024-03-25T18:12:59.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:59.071+0000 7fe457fff640 1 -- 172.21.15.156:0/3164943734 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7fe4500524b0 con 0x7fe4600b1730 2024-03-25T18:12:59.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:59.071+0000 7fe457fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:59.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:59.071+0000 7fe457fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:59.072 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:12:59.072 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:59.075 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:59.073+0000 7fe46619d640 10 librados: watch_flush enter 2024-03-25T18:12:59.075 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:59.073+0000 7fe46619d640 10 librados: watch_flush exit 2024-03-25T18:12:59.075 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:59.073+0000 7fe46619d640 20 client.4434.objecter shutdown clearing up homeless session... 2024-03-25T18:12:59.075 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:59.073+0000 7fe46619d640 10 client.4434.objecter successfully canceled tick 2024-03-25T18:12:59.075 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:59.073+0000 7fe46619d640 1 -- 172.21.15.156:0/3164943734 >> v1:172.21.15.194:6800/2946427713 conn(0x7fe44006de90 legacy=0x7fe440070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:59.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:59.073+0000 7fe46619d640 10 monclient: shutdown 2024-03-25T18:12:59.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:59.073+0000 7fe46619d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:59.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:59.073+0000 7fe46619d640 1 -- 172.21.15.156:0/3164943734 >> v1:172.21.15.156:6789/0 conn(0x7fe4600b1730 legacy=0x7fe460131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:59.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:59.074+0000 7fe46619d640 1 -- 172.21.15.156:0/3164943734 shutdown_connections 2024-03-25T18:12:59.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:59.074+0000 7fe46619d640 1 -- 172.21.15.156:0/3164943734 >> 172.21.15.156:0/3164943734 conn(0x7fe4600617a0 msgr2=0x7fe460061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:59.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:59.074+0000 7fe46619d640 1 -- 172.21.15.156:0/3164943734 shutdown_connections 2024-03-25T18:12:59.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:59.074+0000 7fe46619d640 1 -- 172.21.15.156:0/3164943734 wait complete. 2024-03-25T18:12:59.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:12:59.074+0000 7fe46619d640 1 librados: shutdown 2024-03-25T18:12:59.088 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:03.961 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:05:09.951 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:03.966 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:05:09.956 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:03.998 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:05:09.988 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:04.016 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:05:09.983 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:04.029 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:05:09.996 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:04.033 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:05:10.023 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:04.051 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:05:10.042 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:04.058 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:05:10.025 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:04.079 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:05:10.069 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:05.894 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:06.008 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.007+0000 7f11e2243640 10 monclient: get_monmap_and_config 2024-03-25T18:13:06.008 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.007+0000 7f11e2243640 10 monclient: build_initial_monmap 2024-03-25T18:13:06.008 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.007+0000 7f11e2243640 10 monclient: monmap: 2024-03-25T18:13:06.008 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:13:06.008 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:06.008 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:13:06.008111+0000 2024-03-25T18:13:06.008 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:13:06.008111+0000 2024-03-25T18:13:06.008 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:06.008 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:06.008 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:13:06.008 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:13:06.008 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:13:06.009 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:06.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.007+0000 7f11e2243640 1 Processor -- start 2024-03-25T18:13:06.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.008+0000 7f11e2243640 1 -- start start 2024-03-25T18:13:06.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.008+0000 7f11e2243640 10 monclient: init 2024-03-25T18:13:06.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.008+0000 7f11e2243640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:06.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.008+0000 7f11e2243640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:13:06.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.008+0000 7f11e2243640 10 monclient(hunting): picked mon.noname-b con 0x7f11dc0a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:13:06.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.008+0000 7f11e2243640 10 monclient(hunting): picked mon.noname-a con 0x7f11dc13f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:13:06.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.008+0000 7f11e2243640 10 monclient(hunting): picked mon.noname-c con 0x7f11dc1430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:13:06.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.008+0000 7f11e2243640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11dc142520 con 0x7f11dc13f710 2024-03-25T18:13:06.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.008+0000 7f11e2243640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11dc0b28a0 con 0x7f11dc1430c0 2024-03-25T18:13:06.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.008+0000 7f11e2243640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11dc0b2a30 con 0x7f11dc0a98f0 2024-03-25T18:13:06.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.008+0000 7f11e2243640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:06.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.008+0000 7f11e2243640 10 monclient(hunting): authenticate will time out at 7267.319482s 2024-03-25T18:13:06.031 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.030+0000 7f11daffd640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f11dc13f710 0x7f11dc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:53226/0 (socket says 172.21.15.156:53226) 2024-03-25T18:13:06.031 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.030+0000 7f11daffd640 1 -- 172.21.15.156:0/389576615 learned_addr learned my addr 172.21.15.156:0/389576615 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:13:06.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.041+0000 7f11da7fc640 1 -- 172.21.15.156:0/389576615 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 91710101 0 0) 0x7f11dc0b2a30 con 0x7f11dc0a98f0 2024-03-25T18:13:06.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.041+0000 7f11da7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:06.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.041+0000 7f11da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:06.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.041+0000 7f11da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:06.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.041+0000 7f11da7fc640 10 monclient(hunting): my global_id is 4445 2024-03-25T18:13:06.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.041+0000 7f11da7fc640 1 -- 172.21.15.156:0/389576615 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11c4003520 con 0x7f11dc0a98f0 2024-03-25T18:13:06.046 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.045+0000 7f11da7fc640 1 -- 172.21.15.156:0/389576615 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3752421399 0 0) 0x7f11dc0b28a0 con 0x7f11dc1430c0 2024-03-25T18:13:06.047 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.045+0000 7f11da7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:06.047 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.045+0000 7f11da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:06.047 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.045+0000 7f11da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:06.047 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.045+0000 7f11da7fc640 10 monclient(hunting): my global_id is 4438 2024-03-25T18:13:06.047 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.045+0000 7f11da7fc640 1 -- 172.21.15.156:0/389576615 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11dc0b2a30 con 0x7f11dc1430c0 2024-03-25T18:13:06.050 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.048+0000 7f11da7fc640 1 -- 172.21.15.156:0/389576615 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2844514382 0 0) 0x7f11c4003520 con 0x7f11dc0a98f0 2024-03-25T18:13:06.050 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.048+0000 7f11da7fc640 1 -- 172.21.15.156:0/389576615 >> v1:172.21.15.194:6789/0 conn(0x7f11dc1430c0 legacy=0x7f11dc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:06.050 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.048+0000 7f11da7fc640 1 -- 172.21.15.156:0/389576615 >> v1:172.21.15.156:6789/0 conn(0x7f11dc13f710 legacy=0x7f11dc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:06.050 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.048+0000 7f11da7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:13:06.050 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.048+0000 7f11da7fc640 1 monclient: found mon.noname-b 2024-03-25T18:13:06.050 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.049+0000 7f11da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:06.050 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.049+0000 7f11da7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.156:6790/0 2024-03-25T18:13:06.050 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.049+0000 7f11da7fc640 1 -- 172.21.15.156:0/389576615 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11dc0b44c0 con 0x7f11dc0a98f0 2024-03-25T18:13:06.050 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.049+0000 7f11da7fc640 10 monclient: _finish_auth 0 2024-03-25T18:13:06.050 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.049+0000 7f11da7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:13:06.050 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.049+0000 7f11da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:06.050 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.049+0000 7f11da7fc640 1 -- 172.21.15.156:0/389576615 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f11cc001690 con 0x7f11dc0a98f0 2024-03-25T18:13:06.050 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.049+0000 7f11e2243640 5 monclient: authenticate success, global_id 4445 2024-03-25T18:13:06.051 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.049+0000 7f11e2243640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:06.051 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.049+0000 7f11da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:06.051 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.049+0000 7f11da7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:06.051 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.049+0000 7f11da7fc640 10 monclient: dump: 2024-03-25T18:13:06.051 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:06.051 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:06.051 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:06.051 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:06.051 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:06.051 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:06.051 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:06.051 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:06.051 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:06.051 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:06.052 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.049+0000 7f11e2243640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:06.068 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.067+0000 7f11da7fc640 1 -- 172.21.15.156:0/389576615 <== mon.2 v1:172.21.15.156:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11cc0017f0 con 0x7f11dc0a98f0 2024-03-25T18:13:06.068 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.067+0000 7f11da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:06.068 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.067+0000 7f11da7fc640 1 -- 172.21.15.156:0/389576615 <== mon.2 v1:172.21.15.156:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f11cc001c00 con 0x7f11dc0a98f0 2024-03-25T18:13:06.069 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.067+0000 7f11da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:06.069 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.067+0000 7f11da7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:06.069 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.067+0000 7f11da7fc640 10 monclient: dump: 2024-03-25T18:13:06.069 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:06.069 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:06.069 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:06.069 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:06.069 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:06.069 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:06.069 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:06.069 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:06.069 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:06.069 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:06.070 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.067+0000 7f11e2243640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:06.070 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.067+0000 7f11e2243640 4 set_mon_vals no callback set 2024-03-25T18:13:06.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.068+0000 7f11e2243640 10 monclient: shutdown 2024-03-25T18:13:06.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.068+0000 7f11e2243640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:06.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.068+0000 7f11e2243640 1 -- 172.21.15.156:0/389576615 >> v1:172.21.15.156:6790/0 conn(0x7f11dc0a98f0 legacy=0x7f11dc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:06.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.068+0000 7f11e2243640 1 -- 172.21.15.156:0/389576615 shutdown_connections 2024-03-25T18:13:06.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.068+0000 7f11e2243640 1 -- 172.21.15.156:0/389576615 >> 172.21.15.156:0/389576615 conn(0x7f11dc0b7ea0 msgr2=0x7f11dc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:06.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.068+0000 7f11e2243640 1 -- 172.21.15.156:0/389576615 shutdown_connections 2024-03-25T18:13:06.073 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.068+0000 7f11e2243640 1 -- 172.21.15.156:0/389576615 wait complete. 2024-03-25T18:13:06.073 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.069+0000 7f11e2243640 10 monclient: build_initial_monmap 2024-03-25T18:13:06.073 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.069+0000 7f11e2243640 10 monclient: monmap: 2024-03-25T18:13:06.073 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:13:06.073 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:06.073 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:13:06.073 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:13:06.073 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:06.073 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:13:06.073 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:13:06.073 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:13:06.073 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:13:06.073 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:06.073 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.069+0000 7f11e2243640 1 librados: starting msgr at 2024-03-25T18:13:06.073 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.069+0000 7f11e2243640 1 librados: starting objecter 2024-03-25T18:13:06.074 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.069+0000 7f11e2243640 1 Processor -- start 2024-03-25T18:13:06.074 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.069+0000 7f11e2243640 1 -- start start 2024-03-25T18:13:06.074 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.069+0000 7f11e2243640 1 librados: setting wanted keys 2024-03-25T18:13:06.074 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.069+0000 7f11e2243640 1 librados: calling monclient init 2024-03-25T18:13:06.074 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.069+0000 7f11e2243640 10 monclient: init 2024-03-25T18:13:06.074 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.070+0000 7f11e2243640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:06.074 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.070+0000 7f11e2243640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:13:06.074 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.070+0000 7f11e2243640 10 monclient(hunting): picked mon.noname-c con 0x7f11dc0a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:13:06.074 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.070+0000 7f11e2243640 10 monclient(hunting): picked mon.noname-a con 0x7f11dc13f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:13:06.074 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.070+0000 7f11e2243640 10 monclient(hunting): picked mon.noname-b con 0x7f11dc1430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:13:06.074 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.070+0000 7f11e2243640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11dc1db7d0 con 0x7f11dc13f710 2024-03-25T18:13:06.074 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.070+0000 7f11e2243640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11dc0b1420 con 0x7f11dc1430c0 2024-03-25T18:13:06.074 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.070+0000 7f11e2243640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11dc0b15b0 con 0x7f11dc0a98f0 2024-03-25T18:13:06.075 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.070+0000 7f11e2243640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:06.075 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.070+0000 7f11e2243640 10 monclient(hunting): authenticate will time out at 7267.381343s 2024-03-25T18:13:06.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.079+0000 7f11dbfff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f11dc1430c0 0x7f11dc1d4350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:50184/0 (socket says 172.21.15.156:50184) 2024-03-25T18:13:06.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.079+0000 7f11dbfff640 1 -- 172.21.15.156:0/398389936 learned_addr learned my addr 172.21.15.156:0/398389936 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:13:06.081 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.080+0000 7f11c3fff640 10 client.?.objecter ms_handle_connect 0x7f11dc1430c0 2024-03-25T18:13:06.081 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.080+0000 7f11c3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:06.084 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.083+0000 7f11c3fff640 10 client.?.objecter ms_handle_connect 0x7f11dc0a98f0 2024-03-25T18:13:06.084 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.083+0000 7f11c3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:06.089 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.088+0000 7f11c3fff640 10 client.?.objecter ms_handle_connect 0x7f11dc13f710 2024-03-25T18:13:06.089 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.088+0000 7f11c3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:06.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.096+0000 7f11c3fff640 1 -- 172.21.15.156:0/398389936 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2887219211 0 0) 0x7f11dc1db7d0 con 0x7f11dc13f710 2024-03-25T18:13:06.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.096+0000 7f11c3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:06.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.096+0000 7f11c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:06.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.096+0000 7f11c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:06.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.096+0000 7f11c3fff640 10 monclient(hunting): my global_id is 4440 2024-03-25T18:13:06.097 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.096+0000 7f11c3fff640 1 -- 172.21.15.156:0/398389936 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11c8004eb0 con 0x7f11dc13f710 2024-03-25T18:13:06.105 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.104+0000 7f11c3fff640 1 -- 172.21.15.156:0/398389936 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533715681 0 0) 0x7f11dc0b1420 con 0x7f11dc1430c0 2024-03-25T18:13:06.105 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.104+0000 7f11c3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:06.105 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.104+0000 7f11c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:06.105 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.104+0000 7f11c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:06.106 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.104+0000 7f11c3fff640 10 monclient(hunting): my global_id is 4441 2024-03-25T18:13:06.106 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.104+0000 7f11c3fff640 1 -- 172.21.15.156:0/398389936 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11dc1db7d0 con 0x7f11dc1430c0 2024-03-25T18:13:06.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.109+0000 7f11dbfff640 1 -- 172.21.15.156:0/398389936 >> v1:172.21.15.194:6789/0 conn(0x7f11dc1430c0 legacy=0x7f11dc1d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T18:13:06.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.109+0000 7f11dbfff640 1 -- 172.21.15.156:0/398389936 >> v1:172.21.15.194:6789/0 conn(0x7f11dc1430c0 legacy=0x7f11dc1d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:13:06.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.109+0000 7f11dbfff640 1 --1- 172.21.15.156:0/398389936 >> v1:172.21.15.194:6789/0 conn(0x7f11dc1430c0 0x7f11dc1d4350 :-1 s=OPENED pgs=33555 cs=1 l=1).handle_message read tag failed 2024-03-25T18:13:06.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.109+0000 7f11dbfff640 1 --1- 172.21.15.156:0/398389936 >> v1:172.21.15.194:6789/0 conn(0x7f11dc1430c0 0x7f11dc1d4350 :-1 s=OPENED pgs=33555 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:13:06.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.109+0000 7f11c3fff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.194:6789/0 2024-03-25T18:13:06.113 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.112+0000 7f11c3fff640 1 -- 172.21.15.156:0/398389936 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4209922446 0 0) 0x7f11dc0b15b0 con 0x7f11dc0a98f0 2024-03-25T18:13:06.113 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.112+0000 7f11c3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:06.113 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.112+0000 7f11c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:06.113 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.112+0000 7f11c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:06.113 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.112+0000 7f11c3fff640 10 monclient(hunting): my global_id is 4448 2024-03-25T18:13:06.113 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.112+0000 7f11c3fff640 1 -- 172.21.15.156:0/398389936 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11dc0b1420 con 0x7f11dc0a98f0 2024-03-25T18:13:06.122 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.121+0000 7f11c3fff640 1 -- 172.21.15.156:0/398389936 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 345232284 0 0) 0x7f11c8004eb0 con 0x7f11dc13f710 2024-03-25T18:13:06.122 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.121+0000 7f11c3fff640 1 -- 172.21.15.156:0/398389936 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11dc0b15b0 con 0x7f11dc13f710 2024-03-25T18:13:06.122 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.121+0000 7f11c3fff640 1 -- 172.21.15.156:0/398389936 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f11d00018a0 con 0x7f11dc13f710 2024-03-25T18:13:06.122 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.121+0000 7f11c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:06.122 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.121+0000 7f11c3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:06.122 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.121+0000 7f11c3fff640 10 monclient(hunting): dump: 2024-03-25T18:13:06.122 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:06.122 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:06.123 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:06.123 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:06.123 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:06.123 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:06.123 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:06.123 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:06.123 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:06.123 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:06.123 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.121+0000 7f11c3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:06.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.131+0000 7f11c3fff640 1 -- 172.21.15.156:0/398389936 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2271512844 0 0) 0x7f11dc0b15b0 con 0x7f11dc13f710 2024-03-25T18:13:06.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.131+0000 7f11c3fff640 1 -- 172.21.15.156:0/398389936 >> v1:172.21.15.156:6790/0 conn(0x7f11dc0a98f0 legacy=0x7f11dc1daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:06.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.131+0000 7f11c3fff640 1 -- 172.21.15.156:0/398389936 >> v1:172.21.15.194:6789/0 conn(0x7f11dc1430c0 legacy=0x7f11dc1d4350 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:13:06.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.131+0000 7f11c3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:13:06.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.131+0000 7f11c3fff640 1 monclient: found mon.a 2024-03-25T18:13:06.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.131+0000 7f11c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:06.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.131+0000 7f11c3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:13:06.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.131+0000 7f11c3fff640 1 -- 172.21.15.156:0/398389936 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11dc0b1740 con 0x7f11dc13f710 2024-03-25T18:13:06.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.131+0000 7f11c3fff640 10 monclient: _finish_auth 0 2024-03-25T18:13:06.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.131+0000 7f11c3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:13:06.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.131+0000 7f11c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:06.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.131+0000 7f11e2243640 5 monclient: authenticate success, global_id 4440 2024-03-25T18:13:06.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.131+0000 7f11e2243640 10 monclient: _renew_subs 2024-03-25T18:13:06.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.131+0000 7f11e2243640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:13:06.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.131+0000 7f11e2243640 1 -- 172.21.15.156:0/398389936 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11dc1db1a0 con 0x7f11dc13f710 2024-03-25T18:13:06.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.131+0000 7f11e2243640 10 client.4440.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:06.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.131+0000 7f11e2243640 10 monclient: _renew_subs 2024-03-25T18:13:06.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.132+0000 7f11e2243640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:13:06.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.132+0000 7f11e2243640 1 -- 172.21.15.156:0/398389936 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11dc0b08f0 con 0x7f11dc13f710 2024-03-25T18:13:06.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.132+0000 7f11e2243640 1 librados: init done 2024-03-25T18:13:06.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.132+0000 7f11e2243640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:06.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.132+0000 7f11e2243640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:06.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.132+0000 7f11e2243640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:13:06.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.132+0000 7f11e2243640 1 -- 172.21.15.156:0/398389936 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11dc0627b0 con 0x7f11dc13f710 2024-03-25T18:13:06.136 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.135+0000 7f11c3fff640 1 -- 172.21.15.156:0/398389936 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11d0001d10 con 0x7f11dc13f710 2024-03-25T18:13:06.136 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.135+0000 7f11c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:06.136 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.135+0000 7f11d9ffb640 4 set_mon_vals no callback set 2024-03-25T18:13:06.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.434+0000 7f11c3fff640 1 -- 172.21.15.156:0/398389936 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f11d0004980 con 0x7f11dc13f710 2024-03-25T18:13:06.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.434+0000 7f11c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:06.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.434+0000 7f11c3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:06.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.434+0000 7f11c3fff640 10 monclient: dump: 2024-03-25T18:13:06.435 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:06.435 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:06.435 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:06.435 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:06.435 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:06.435 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:06.436 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:06.436 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:06.436 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:06.436 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:06.436 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.435+0000 7f11c3fff640 1 -- 172.21.15.156:0/398389936 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f11d0004ba0 con 0x7f11dc13f710 2024-03-25T18:13:06.437 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.436+0000 7f11db7fe640 10 monclient: get_auth_request con 0x7f11c806dd30 auth_method 0 2024-03-25T18:13:06.437 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.436+0000 7f11c3fff640 10 client.4440.objecter ms_handle_connect 0x7f11c806dd30 2024-03-25T18:13:06.445 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.444+0000 7f11c3fff640 1 -- 172.21.15.156:0/398389936 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f11d0054760 con 0x7f11dc13f710 2024-03-25T18:13:06.445 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.444+0000 7f11c3fff640 10 client.4440.objecter ms_dispatch 0x7f11dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:06.445 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.444+0000 7f11c3fff640 3 client.4440.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:06.445 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.444+0000 7f11c3fff640 3 client.4440.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:06.445 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.444+0000 7f11c3fff640 20 client.4440.objecter dump_active .. 0 homeless 2024-03-25T18:13:06.445 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.444+0000 7f11c3fff640 1 -- 172.21.15.156:0/398389936 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f11d00500a0 con 0x7f11dc13f710 2024-03-25T18:13:06.445 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.444+0000 7f11c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:06.445 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.444+0000 7f11c3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:06.666 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.665+0000 7f11e2243640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:06.666 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.665+0000 7f11e2243640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:06.666 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.665+0000 7f11e2243640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:13:06.666 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.665+0000 7f11e2243640 1 -- 172.21.15.156:0/398389936 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11dc0abb10 con 0x7f11dc13f710 2024-03-25T18:13:06.691 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.690+0000 7f11c3fff640 1 -- 172.21.15.156:0/398389936 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f11d00521a0 con 0x7f11dc13f710 2024-03-25T18:13:06.691 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.690+0000 7f11c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:06.691 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.690+0000 7f11c3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:06.691 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:13:06.692 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:06.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.692+0000 7f11e2243640 10 librados: watch_flush enter 2024-03-25T18:13:06.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.692+0000 7f11e2243640 10 librados: watch_flush exit 2024-03-25T18:13:06.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.692+0000 7f11e2243640 20 client.4440.objecter shutdown clearing up homeless session... 2024-03-25T18:13:06.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.692+0000 7f11e2243640 10 client.4440.objecter successfully canceled tick 2024-03-25T18:13:06.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.692+0000 7f11e2243640 1 -- 172.21.15.156:0/398389936 >> v1:172.21.15.194:6800/2946427713 conn(0x7f11c806dd30 legacy=0x7f11c80701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:06.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.693+0000 7f11e2243640 10 monclient: shutdown 2024-03-25T18:13:06.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.693+0000 7f11e2243640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:06.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.693+0000 7f11e2243640 1 -- 172.21.15.156:0/398389936 >> v1:172.21.15.156:6789/0 conn(0x7f11dc13f710 legacy=0x7f11dc0b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:06.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.693+0000 7f11e2243640 1 -- 172.21.15.156:0/398389936 shutdown_connections 2024-03-25T18:13:06.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.693+0000 7f11e2243640 1 -- 172.21.15.156:0/398389936 >> 172.21.15.156:0/398389936 conn(0x7f11dc0b7ea0 msgr2=0x7f11dc13c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:06.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.693+0000 7f11e2243640 1 -- 172.21.15.156:0/398389936 shutdown_connections 2024-03-25T18:13:06.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.693+0000 7f11e2243640 1 -- 172.21.15.156:0/398389936 wait complete. 2024-03-25T18:13:06.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:06.693+0000 7f11e2243640 1 librados: shutdown 2024-03-25T18:13:06.705 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:13.511 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:13.627 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.626+0000 7ff4ae255640 10 monclient: get_monmap_and_config 2024-03-25T18:13:13.627 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.626+0000 7ff4ae255640 10 monclient: build_initial_monmap 2024-03-25T18:13:13.627 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.626+0000 7ff4ae255640 10 monclient: monmap: 2024-03-25T18:13:13.627 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:13:13.627 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:13.627 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:13:13.627524+0000 2024-03-25T18:13:13.628 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:13:13.627524+0000 2024-03-25T18:13:13.628 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:13.628 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:13.628 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:13:13.628 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:13:13.628 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:13:13.628 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:13.629 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.627+0000 7ff4ae255640 1 Processor -- start 2024-03-25T18:13:13.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.627+0000 7ff4ae255640 1 -- start start 2024-03-25T18:13:13.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.627+0000 7ff4ae255640 10 monclient: init 2024-03-25T18:13:13.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.628+0000 7ff4ae255640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:13.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.628+0000 7ff4ae255640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:13.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.628+0000 7ff4ae255640 10 monclient(hunting): picked mon.noname-b con 0x7ff4a80b4d60 addr v1:172.21.15.156:6790/0 2024-03-25T18:13:13.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.628+0000 7ff4ae255640 10 monclient(hunting): picked mon.noname-c con 0x7ff4a80b1730 addr v1:172.21.15.194:6789/0 2024-03-25T18:13:13.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.628+0000 7ff4ae255640 10 monclient(hunting): picked mon.noname-a con 0x7ff4a814be90 addr v1:172.21.15.156:6789/0 2024-03-25T18:13:13.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.628+0000 7ff4ae255640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4a80af940 con 0x7ff4a814be90 2024-03-25T18:13:13.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.628+0000 7ff4ae255640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4a80b0a70 con 0x7ff4a80b1730 2024-03-25T18:13:13.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.628+0000 7ff4ae255640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4a80b0d50 con 0x7ff4a80b4d60 2024-03-25T18:13:13.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.628+0000 7ff4ae255640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:13.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.628+0000 7ff4ae255640 10 monclient(hunting): authenticate will time out at 7274.939086s 2024-03-25T18:13:13.637 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.636+0000 7ff4a7fff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7ff4a814be90 0x7ff4a814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:41050/0 (socket says 172.21.15.156:41050) 2024-03-25T18:13:13.637 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.636+0000 7ff4a7fff640 1 -- 172.21.15.156:0/1998916158 learned_addr learned my addr 172.21.15.156:0/1998916158 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:13:13.644 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.643+0000 7ff4a67fc640 1 -- 172.21.15.156:0/1998916158 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3070245755 0 0) 0x7ff4a80af940 con 0x7ff4a814be90 2024-03-25T18:13:13.644 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.644+0000 7ff4a67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:13.645 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.644+0000 7ff4a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:13.645 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.644+0000 7ff4a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:13.645 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.644+0000 7ff4a67fc640 10 monclient(hunting): my global_id is 4443 2024-03-25T18:13:13.645 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.644+0000 7ff4a67fc640 1 -- 172.21.15.156:0/1998916158 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff490003520 con 0x7ff4a814be90 2024-03-25T18:13:13.657 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.656+0000 7ff4a67fc640 1 -- 172.21.15.156:0/1998916158 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302277819 0 0) 0x7ff4a80b0a70 con 0x7ff4a80b1730 2024-03-25T18:13:13.657 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.656+0000 7ff4a67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:13.657 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.656+0000 7ff4a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:13.657 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.656+0000 7ff4a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:13.657 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.656+0000 7ff4a67fc640 10 monclient(hunting): my global_id is 4444 2024-03-25T18:13:13.657 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.656+0000 7ff4a67fc640 1 -- 172.21.15.156:0/1998916158 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4a80af940 con 0x7ff4a80b1730 2024-03-25T18:13:13.658 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.657+0000 7ff4a67fc640 1 -- 172.21.15.156:0/1998916158 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3655794017 0 0) 0x7ff490003520 con 0x7ff4a814be90 2024-03-25T18:13:13.658 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.657+0000 7ff4a67fc640 1 -- 172.21.15.156:0/1998916158 >> v1:172.21.15.156:6790/0 conn(0x7ff4a80b4d60 legacy=0x7ff4a813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:13.658 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.657+0000 7ff4a67fc640 1 -- 172.21.15.156:0/1998916158 >> v1:172.21.15.194:6789/0 conn(0x7ff4a80b1730 legacy=0x7ff4a80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:13.658 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.657+0000 7ff4a67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:13:13.659 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.657+0000 7ff4a67fc640 1 monclient: found mon.noname-a 2024-03-25T18:13:13.659 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.657+0000 7ff4a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:13.659 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.657+0000 7ff4a67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:13:13.659 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.657+0000 7ff4a67fc640 1 -- 172.21.15.156:0/1998916158 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a80b1030 con 0x7ff4a814be90 2024-03-25T18:13:13.659 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.657+0000 7ff4a67fc640 10 monclient: _finish_auth 0 2024-03-25T18:13:13.659 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.657+0000 7ff4a67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:13:13.659 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.657+0000 7ff4a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:13.659 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.657+0000 7ff4a67fc640 1 -- 172.21.15.156:0/1998916158 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7ff4a0001a10 con 0x7ff4a814be90 2024-03-25T18:13:13.659 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.657+0000 7ff4a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:13.659 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.657+0000 7ff4a67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:13.659 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.657+0000 7ff4a67fc640 10 monclient: dump: 2024-03-25T18:13:13.659 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:13.659 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:13.659 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:13.659 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:13.660 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:13.660 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:13.660 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:13.660 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:13.660 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:13.660 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:13.660 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.657+0000 7ff4ae255640 5 monclient: authenticate success, global_id 4443 2024-03-25T18:13:13.660 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.657+0000 7ff4ae255640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:13.665 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.664+0000 7ff4a67fc640 1 -- 172.21.15.156:0/1998916158 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4a0001b70 con 0x7ff4a814be90 2024-03-25T18:13:13.665 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.664+0000 7ff4a67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:13.665 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.664+0000 7ff4a67fc640 1 -- 172.21.15.156:0/1998916158 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7ff4a0001f80 con 0x7ff4a814be90 2024-03-25T18:13:13.665 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.664+0000 7ff4ae255640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:13.665 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.664+0000 7ff4ae255640 4 set_mon_vals no callback set 2024-03-25T18:13:13.665 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.664+0000 7ff4a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:13.666 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.664+0000 7ff4a67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:13.666 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.664+0000 7ff4a67fc640 10 monclient: dump: 2024-03-25T18:13:13.666 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:13.666 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:13.666 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:13.666 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:13.666 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:13.666 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:13.666 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:13.666 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:13.666 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:13.666 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:13.668 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.665+0000 7ff4ae255640 10 monclient: shutdown 2024-03-25T18:13:13.668 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.665+0000 7ff4ae255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:13.668 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.665+0000 7ff4ae255640 1 -- 172.21.15.156:0/1998916158 >> v1:172.21.15.156:6789/0 conn(0x7ff4a814be90 legacy=0x7ff4a814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:13.669 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.665+0000 7ff4ae255640 1 -- 172.21.15.156:0/1998916158 shutdown_connections 2024-03-25T18:13:13.669 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.665+0000 7ff4ae255640 1 -- 172.21.15.156:0/1998916158 >> 172.21.15.156:0/1998916158 conn(0x7ff4a80617a0 msgr2=0x7ff4a8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:13.669 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.665+0000 7ff4ae255640 1 -- 172.21.15.156:0/1998916158 shutdown_connections 2024-03-25T18:13:13.669 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.665+0000 7ff4ae255640 1 -- 172.21.15.156:0/1998916158 wait complete. 2024-03-25T18:13:13.669 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.665+0000 7ff4ae255640 10 monclient: build_initial_monmap 2024-03-25T18:13:13.669 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.665+0000 7ff4ae255640 10 monclient: monmap: 2024-03-25T18:13:13.669 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:13:13.669 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:13.669 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:13:13.669 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:13:13.669 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:13.669 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:13:13.669 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:13:13.669 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:13:13.669 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:13:13.669 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:13.669 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.666+0000 7ff4ae255640 1 librados: starting msgr at 2024-03-25T18:13:13.670 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.666+0000 7ff4ae255640 1 librados: starting objecter 2024-03-25T18:13:13.670 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.666+0000 7ff4ae255640 1 Processor -- start 2024-03-25T18:13:13.670 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.666+0000 7ff4ae255640 1 -- start start 2024-03-25T18:13:13.670 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.666+0000 7ff4ae255640 1 librados: setting wanted keys 2024-03-25T18:13:13.670 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.666+0000 7ff4ae255640 1 librados: calling monclient init 2024-03-25T18:13:13.670 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.666+0000 7ff4ae255640 10 monclient: init 2024-03-25T18:13:13.670 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.667+0000 7ff4ae255640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:13.670 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.667+0000 7ff4ae255640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:13:13.670 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.667+0000 7ff4ae255640 10 monclient(hunting): picked mon.noname-b con 0x7ff4a80b1730 addr v1:172.21.15.194:6789/0 2024-03-25T18:13:13.670 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.667+0000 7ff4ae255640 10 monclient(hunting): picked mon.noname-c con 0x7ff4a80b4d60 addr v1:172.21.15.156:6790/0 2024-03-25T18:13:13.670 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.667+0000 7ff4ae255640 10 monclient(hunting): picked mon.noname-a con 0x7ff4a814be90 addr v1:172.21.15.156:6789/0 2024-03-25T18:13:13.670 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.667+0000 7ff4ae255640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4a8142710 con 0x7ff4a814be90 2024-03-25T18:13:13.670 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.667+0000 7ff4ae255640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4a8142410 con 0x7ff4a80b1730 2024-03-25T18:13:13.670 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.667+0000 7ff4ae255640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4a8140d00 con 0x7ff4a80b4d60 2024-03-25T18:13:13.671 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.667+0000 7ff4ae255640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:13.671 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.667+0000 7ff4ae255640 10 monclient(hunting): authenticate will time out at 7274.978217s 2024-03-25T18:13:13.682 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.681+0000 7ff4a77fe640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7ff4a80b1730 0x7ff4a81405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:37658/0 (socket says 172.21.15.156:37658) 2024-03-25T18:13:13.682 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.681+0000 7ff4a77fe640 1 -- 172.21.15.156:0/204520559 learned_addr learned my addr 172.21.15.156:0/204520559 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:13:13.685 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.684+0000 7ff48ffff640 10 client.?.objecter ms_handle_connect 0x7ff4a80b1730 2024-03-25T18:13:13.685 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.684+0000 7ff48ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:13.688 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.687+0000 7ff48ffff640 10 client.?.objecter ms_handle_connect 0x7ff4a80b4d60 2024-03-25T18:13:13.688 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.687+0000 7ff48ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:13.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.694+0000 7ff48ffff640 10 client.?.objecter ms_handle_connect 0x7ff4a814be90 2024-03-25T18:13:13.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.694+0000 7ff48ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:13.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.699+0000 7ff48ffff640 1 -- 172.21.15.156:0/204520559 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 944497615 0 0) 0x7ff4a8140d00 con 0x7ff4a80b4d60 2024-03-25T18:13:13.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.699+0000 7ff48ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:13.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.699+0000 7ff48ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:13.701 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.699+0000 7ff48ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:13.701 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.700+0000 7ff48ffff640 10 monclient(hunting): my global_id is 4451 2024-03-25T18:13:13.701 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.700+0000 7ff48ffff640 1 -- 172.21.15.156:0/204520559 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff494004eb0 con 0x7ff4a80b4d60 2024-03-25T18:13:13.702 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.701+0000 7ff48ffff640 1 -- 172.21.15.156:0/204520559 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2523164512 0 0) 0x7ff4a8142410 con 0x7ff4a80b1730 2024-03-25T18:13:13.702 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.701+0000 7ff48ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:13.703 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.701+0000 7ff48ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:13.703 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.701+0000 7ff48ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:13.703 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.701+0000 7ff48ffff640 10 monclient(hunting): my global_id is 4447 2024-03-25T18:13:13.703 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.701+0000 7ff48ffff640 1 -- 172.21.15.156:0/204520559 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4a8140d00 con 0x7ff4a80b1730 2024-03-25T18:13:13.708 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.707+0000 7ff48ffff640 1 -- 172.21.15.156:0/204520559 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957890421 0 0) 0x7ff4a8142710 con 0x7ff4a814be90 2024-03-25T18:13:13.708 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.707+0000 7ff48ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:13.708 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.707+0000 7ff48ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:13.708 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.707+0000 7ff48ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:13.708 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.707+0000 7ff48ffff640 10 monclient(hunting): my global_id is 4446 2024-03-25T18:13:13.708 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.707+0000 7ff48ffff640 1 -- 172.21.15.156:0/204520559 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4a8142410 con 0x7ff4a814be90 2024-03-25T18:13:13.718 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.717+0000 7ff48ffff640 1 -- 172.21.15.156:0/204520559 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 165242483 0 0) 0x7ff494004eb0 con 0x7ff4a80b4d60 2024-03-25T18:13:13.718 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.717+0000 7ff48ffff640 1 -- 172.21.15.156:0/204520559 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4a8142710 con 0x7ff4a80b4d60 2024-03-25T18:13:13.718 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.717+0000 7ff48ffff640 1 -- 172.21.15.156:0/204520559 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7ff49c001930 con 0x7ff4a80b4d60 2024-03-25T18:13:13.718 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.717+0000 7ff48ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:13.718 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.717+0000 7ff48ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:13.718 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.717+0000 7ff48ffff640 10 monclient(hunting): dump: 2024-03-25T18:13:13.718 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:13.718 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:13.718 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:13.718 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:13.718 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:13.718 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:13.718 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:13.719 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:13.719 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:13.719 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:13.719 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.717+0000 7ff48ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:13.722 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.721+0000 7ff48ffff640 1 -- 172.21.15.156:0/204520559 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3431570222 0 0) 0x7ff4a8140d00 con 0x7ff4a80b1730 2024-03-25T18:13:13.722 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.721+0000 7ff48ffff640 1 -- 172.21.15.156:0/204520559 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff494004eb0 con 0x7ff4a80b1730 2024-03-25T18:13:13.722 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.721+0000 7ff48ffff640 1 -- 172.21.15.156:0/204520559 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7ff4980019b0 con 0x7ff4a80b1730 2024-03-25T18:13:13.722 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.721+0000 7ff48ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:13.723 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.721+0000 7ff48ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:13.723 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.721+0000 7ff48ffff640 10 monclient(hunting): dump: 2024-03-25T18:13:13.723 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:13.723 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:13.723 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:13.723 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:13.723 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:13.723 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:13.723 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:13.723 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:13.723 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:13.723 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:13.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.731+0000 7ff48ffff640 1 -- 172.21.15.156:0/204520559 <== mon.2 v1:172.21.15.156:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2130653197 0 0) 0x7ff4a8142710 con 0x7ff4a80b4d60 2024-03-25T18:13:13.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.731+0000 7ff48ffff640 1 -- 172.21.15.156:0/204520559 >> v1:172.21.15.194:6789/0 conn(0x7ff4a80b1730 legacy=0x7ff4a81405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:13.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.731+0000 7ff48ffff640 1 -- 172.21.15.156:0/204520559 >> v1:172.21.15.156:6789/0 conn(0x7ff4a814be90 legacy=0x7ff4a81e2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:13.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.731+0000 7ff48ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:13:13.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.731+0000 7ff48ffff640 1 monclient: found mon.c 2024-03-25T18:13:13.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.731+0000 7ff48ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:13.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.731+0000 7ff48ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:13:13.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.731+0000 7ff48ffff640 1 -- 172.21.15.156:0/204520559 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a8139000 con 0x7ff4a80b4d60 2024-03-25T18:13:13.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.731+0000 7ff48ffff640 10 monclient: _finish_auth 0 2024-03-25T18:13:13.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.731+0000 7ff48ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:13:13.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.731+0000 7ff48ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:13.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.731+0000 7ff4ae255640 5 monclient: authenticate success, global_id 4451 2024-03-25T18:13:13.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.731+0000 7ff4ae255640 10 monclient: _renew_subs 2024-03-25T18:13:13.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.731+0000 7ff4ae255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:13:13.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.731+0000 7ff4ae255640 1 -- 172.21.15.156:0/204520559 --> v1:172.21.15.156:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4a8141c30 con 0x7ff4a80b4d60 2024-03-25T18:13:13.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.731+0000 7ff4ae255640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:13.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.731+0000 7ff4ae255640 10 monclient: _renew_subs 2024-03-25T18:13:13.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.731+0000 7ff4ae255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:13:13.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.731+0000 7ff4ae255640 1 -- 172.21.15.156:0/204520559 --> v1:172.21.15.156:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4a81432b0 con 0x7ff4a80b4d60 2024-03-25T18:13:13.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.731+0000 7ff4ae255640 1 librados: init done 2024-03-25T18:13:13.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.732+0000 7ff4ae255640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:13.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.732+0000 7ff4ae255640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:13.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.732+0000 7ff4ae255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:13:13.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:13.732+0000 7ff4ae255640 1 -- 172.21.15.156:0/204520559 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4a81428c0 con 0x7ff4a80b4d60 2024-03-25T18:13:14.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.042+0000 7ff48ffff640 1 -- 172.21.15.156:0/204520559 <== mon.2 v1:172.21.15.156:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff49c003610 con 0x7ff4a80b4d60 2024-03-25T18:13:14.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.042+0000 7ff48ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:14.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.042+0000 7ff48ffff640 1 -- 172.21.15.156:0/204520559 <== mon.2 v1:172.21.15.156:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7ff49c003780 con 0x7ff4a80b4d60 2024-03-25T18:13:14.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.042+0000 7ff48ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:14.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.042+0000 7ff48ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:14.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.042+0000 7ff48ffff640 10 monclient: dump: 2024-03-25T18:13:14.043 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:14.043 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:14.043 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:14.043 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:14.043 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:14.043 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:14.044 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:14.044 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:14.044 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:14.044 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:14.044 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.042+0000 7ff4a5ffb640 4 set_mon_vals no callback set 2024-03-25T18:13:14.044 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.042+0000 7ff48ffff640 1 -- 172.21.15.156:0/204520559 <== mon.2 v1:172.21.15.156:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7ff49c019790 con 0x7ff4a80b4d60 2024-03-25T18:13:14.044 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.043+0000 7ff4a77fe640 10 monclient: get_auth_request con 0x7ff49406df20 auth_method 0 2024-03-25T18:13:14.044 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.043+0000 7ff48ffff640 10 client.4451.objecter ms_handle_connect 0x7ff49406df20 2024-03-25T18:13:14.050 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.049+0000 7ff48ffff640 1 -- 172.21.15.156:0/204520559 <== mon.2 v1:172.21.15.156:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7ff49c0534b0 con 0x7ff4a80b4d60 2024-03-25T18:13:14.050 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.049+0000 7ff48ffff640 10 client.4451.objecter ms_dispatch 0x7ff4a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:14.050 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.049+0000 7ff48ffff640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:14.050 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.049+0000 7ff48ffff640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:14.050 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.049+0000 7ff48ffff640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-25T18:13:14.050 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.049+0000 7ff48ffff640 1 -- 172.21.15.156:0/204520559 <== mon.2 v1:172.21.15.156:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff49c04edf0 con 0x7ff4a80b4d60 2024-03-25T18:13:14.050 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.049+0000 7ff48ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:14.050 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.049+0000 7ff48ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:14.289 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.288+0000 7ff4ae255640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:14.289 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.288+0000 7ff4ae255640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:14.289 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.288+0000 7ff4ae255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:13:14.289 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.288+0000 7ff4ae255640 1 -- 172.21.15.156:0/204520559 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4a80627b0 con 0x7ff4a80b4d60 2024-03-25T18:13:14.395 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.393+0000 7ff48ffff640 1 -- 172.21.15.156:0/204520559 <== mon.2 v1:172.21.15.156:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7ff49c050ef0 con 0x7ff4a80b4d60 2024-03-25T18:13:14.395 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.393+0000 7ff48ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:14.395 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.393+0000 7ff48ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:14.395 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:13:14.395 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:14.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.396+0000 7ff4ae255640 10 librados: watch_flush enter 2024-03-25T18:13:14.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.396+0000 7ff4ae255640 10 librados: watch_flush exit 2024-03-25T18:13:14.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.396+0000 7ff4ae255640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-25T18:13:14.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.396+0000 7ff4ae255640 10 client.4451.objecter successfully canceled tick 2024-03-25T18:13:14.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.396+0000 7ff4ae255640 1 -- 172.21.15.156:0/204520559 >> v1:172.21.15.194:6800/2946427713 conn(0x7ff49406df20 legacy=0x7ff4940703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:14.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.396+0000 7ff4ae255640 10 monclient: shutdown 2024-03-25T18:13:14.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.396+0000 7ff4ae255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:14.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.396+0000 7ff4ae255640 1 -- 172.21.15.156:0/204520559 >> v1:172.21.15.156:6790/0 conn(0x7ff4a80b4d60 legacy=0x7ff4a8142ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:14.399 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.396+0000 7ff4ae255640 1 -- 172.21.15.156:0/204520559 shutdown_connections 2024-03-25T18:13:14.399 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.396+0000 7ff4ae255640 1 -- 172.21.15.156:0/204520559 >> 172.21.15.156:0/204520559 conn(0x7ff4a80617a0 msgr2=0x7ff4a80b6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:14.399 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.397+0000 7ff4ae255640 1 -- 172.21.15.156:0/204520559 shutdown_connections 2024-03-25T18:13:14.399 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.397+0000 7ff4ae255640 1 -- 172.21.15.156:0/204520559 wait complete. 2024-03-25T18:13:14.399 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:14.397+0000 7ff4ae255640 1 librados: shutdown 2024-03-25T18:13:14.410 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:21.216 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:21.330 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.328+0000 7f9a2f614640 10 monclient: get_monmap_and_config 2024-03-25T18:13:21.330 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.328+0000 7f9a2f614640 10 monclient: build_initial_monmap 2024-03-25T18:13:21.330 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.328+0000 7f9a2f614640 10 monclient: monmap: 2024-03-25T18:13:21.330 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:13:21.330 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:21.330 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:13:21.329831+0000 2024-03-25T18:13:21.330 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:13:21.329831+0000 2024-03-25T18:13:21.330 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:21.330 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:21.330 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:13:21.330 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:13:21.331 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:13:21.331 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:21.331 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.329+0000 7f9a2f614640 1 Processor -- start 2024-03-25T18:13:21.332 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.329+0000 7f9a2f614640 1 -- start start 2024-03-25T18:13:21.332 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.329+0000 7f9a2f614640 10 monclient: init 2024-03-25T18:13:21.332 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.330+0000 7f9a2f614640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:21.332 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.330+0000 7f9a2f614640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:13:21.332 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.330+0000 7f9a2f614640 10 monclient(hunting): picked mon.noname-b con 0x7f9a280a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:13:21.332 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.330+0000 7f9a2f614640 10 monclient(hunting): picked mon.noname-a con 0x7f9a2813f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:13:21.332 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.330+0000 7f9a2f614640 10 monclient(hunting): picked mon.noname-c con 0x7f9a281430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:13:21.332 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.330+0000 7f9a2f614640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a28142520 con 0x7f9a2813f710 2024-03-25T18:13:21.332 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.330+0000 7f9a2f614640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a280b28a0 con 0x7f9a281430c0 2024-03-25T18:13:21.332 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.330+0000 7f9a2f614640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a280b2a30 con 0x7f9a280a98f0 2024-03-25T18:13:21.332 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.330+0000 7f9a2f614640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:21.332 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.330+0000 7f9a2f614640 10 monclient(hunting): authenticate will time out at 7282.641589s 2024-03-25T18:13:21.342 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.341+0000 7f9a2d3b0640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f9a280a98f0 0x7f9a280a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:36510/0 (socket says 172.21.15.156:36510) 2024-03-25T18:13:21.342 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.341+0000 7f9a2d3b0640 1 -- 172.21.15.156:0/216775654 learned_addr learned my addr 172.21.15.156:0/216775654 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:13:21.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.351+0000 7f9a1ffff640 1 -- 172.21.15.156:0/216775654 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4218488055 0 0) 0x7f9a280b2a30 con 0x7f9a280a98f0 2024-03-25T18:13:21.352 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.351+0000 7f9a1ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:21.352 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.351+0000 7f9a1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:21.352 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.351+0000 7f9a1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:21.352 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.351+0000 7f9a1ffff640 10 monclient(hunting): my global_id is 4454 2024-03-25T18:13:21.352 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.351+0000 7f9a1ffff640 1 -- 172.21.15.156:0/216775654 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a14003520 con 0x7f9a280a98f0 2024-03-25T18:13:21.361 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.360+0000 7f9a1ffff640 1 -- 172.21.15.156:0/216775654 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709497534 0 0) 0x7f9a280b28a0 con 0x7f9a281430c0 2024-03-25T18:13:21.362 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.361+0000 7f9a1ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:21.362 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.361+0000 7f9a1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:21.362 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.361+0000 7f9a1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:21.362 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.361+0000 7f9a1ffff640 10 monclient(hunting): my global_id is 4450 2024-03-25T18:13:21.362 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.361+0000 7f9a1ffff640 1 -- 172.21.15.156:0/216775654 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a280b2a30 con 0x7f9a281430c0 2024-03-25T18:13:21.364 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.363+0000 7f9a1ffff640 1 -- 172.21.15.156:0/216775654 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716431046 0 0) 0x7f9a28142520 con 0x7f9a2813f710 2024-03-25T18:13:21.364 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.363+0000 7f9a1ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:21.364 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.363+0000 7f9a1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:21.364 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.363+0000 7f9a1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:21.364 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.363+0000 7f9a1ffff640 10 monclient(hunting): my global_id is 4449 2024-03-25T18:13:21.364 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.363+0000 7f9a1ffff640 1 -- 172.21.15.156:0/216775654 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a280b28a0 con 0x7f9a2813f710 2024-03-25T18:13:21.366 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.364+0000 7f9a1ffff640 1 -- 172.21.15.156:0/216775654 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3786028838 0 0) 0x7f9a14003520 con 0x7f9a280a98f0 2024-03-25T18:13:21.366 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.364+0000 7f9a1ffff640 1 -- 172.21.15.156:0/216775654 >> v1:172.21.15.194:6789/0 conn(0x7f9a281430c0 legacy=0x7f9a2814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:21.366 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.364+0000 7f9a1ffff640 1 -- 172.21.15.156:0/216775654 >> v1:172.21.15.156:6789/0 conn(0x7f9a2813f710 legacy=0x7f9a281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:21.366 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.364+0000 7f9a1ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:13:21.366 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.364+0000 7f9a1ffff640 1 monclient: found mon.noname-b 2024-03-25T18:13:21.366 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.364+0000 7f9a1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:21.366 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.364+0000 7f9a1ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.156:6790/0 2024-03-25T18:13:21.366 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.364+0000 7f9a1ffff640 1 -- 172.21.15.156:0/216775654 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a280b44c0 con 0x7f9a280a98f0 2024-03-25T18:13:21.366 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.364+0000 7f9a1ffff640 10 monclient: _finish_auth 0 2024-03-25T18:13:21.367 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.364+0000 7f9a1ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:13:21.367 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.364+0000 7f9a1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:21.367 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.364+0000 7f9a1ffff640 1 -- 172.21.15.156:0/216775654 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9a24001690 con 0x7f9a280a98f0 2024-03-25T18:13:21.367 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.364+0000 7f9a2f614640 5 monclient: authenticate success, global_id 4454 2024-03-25T18:13:21.367 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.364+0000 7f9a2f614640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:21.367 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.364+0000 7f9a1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:21.367 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.364+0000 7f9a1ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:21.367 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.364+0000 7f9a1ffff640 10 monclient: dump: 2024-03-25T18:13:21.367 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:21.367 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:21.367 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:21.367 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:21.367 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:21.367 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:21.367 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:21.367 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:21.368 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:21.368 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:21.368 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.364+0000 7f9a2f614640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:21.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.380+0000 7f9a1ffff640 1 -- 172.21.15.156:0/216775654 <== mon.2 v1:172.21.15.156:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a240017f0 con 0x7f9a280a98f0 2024-03-25T18:13:21.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.380+0000 7f9a1ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:21.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.380+0000 7f9a1ffff640 1 -- 172.21.15.156:0/216775654 <== mon.2 v1:172.21.15.156:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9a24001c00 con 0x7f9a280a98f0 2024-03-25T18:13:21.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.380+0000 7f9a2f614640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:21.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.380+0000 7f9a2f614640 4 set_mon_vals no callback set 2024-03-25T18:13:21.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.380+0000 7f9a1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:21.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.380+0000 7f9a1ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:21.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.380+0000 7f9a1ffff640 10 monclient: dump: 2024-03-25T18:13:21.381 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:21.382 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:21.382 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:21.382 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:21.382 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:21.382 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:21.382 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:21.382 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:21.382 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:21.382 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:21.383 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.381+0000 7f9a2f614640 10 monclient: shutdown 2024-03-25T18:13:21.383 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.381+0000 7f9a2f614640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:21.383 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.381+0000 7f9a2f614640 1 -- 172.21.15.156:0/216775654 >> v1:172.21.15.156:6790/0 conn(0x7f9a280a98f0 legacy=0x7f9a280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:21.383 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.381+0000 7f9a2f614640 1 -- 172.21.15.156:0/216775654 shutdown_connections 2024-03-25T18:13:21.384 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.381+0000 7f9a2f614640 1 -- 172.21.15.156:0/216775654 >> 172.21.15.156:0/216775654 conn(0x7f9a280b7ea0 msgr2=0x7f9a280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:21.384 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.381+0000 7f9a2f614640 1 -- 172.21.15.156:0/216775654 shutdown_connections 2024-03-25T18:13:21.384 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.381+0000 7f9a2f614640 1 -- 172.21.15.156:0/216775654 wait complete. 2024-03-25T18:13:21.384 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.382+0000 7f9a2f614640 10 monclient: build_initial_monmap 2024-03-25T18:13:21.384 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.382+0000 7f9a2f614640 10 monclient: monmap: 2024-03-25T18:13:21.384 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:13:21.384 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:21.384 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:13:21.384 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:13:21.384 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:21.384 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:13:21.384 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:13:21.384 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:13:21.384 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:13:21.384 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:21.384 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.382+0000 7f9a2f614640 1 librados: starting msgr at 2024-03-25T18:13:21.384 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.382+0000 7f9a2f614640 1 librados: starting objecter 2024-03-25T18:13:21.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.382+0000 7f9a2f614640 1 Processor -- start 2024-03-25T18:13:21.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.382+0000 7f9a2f614640 1 -- start start 2024-03-25T18:13:21.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.382+0000 7f9a2f614640 1 librados: setting wanted keys 2024-03-25T18:13:21.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.382+0000 7f9a2f614640 1 librados: calling monclient init 2024-03-25T18:13:21.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.382+0000 7f9a2f614640 10 monclient: init 2024-03-25T18:13:21.386 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.383+0000 7f9a2f614640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:21.386 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.383+0000 7f9a2f614640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:21.386 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.383+0000 7f9a2f614640 10 monclient(hunting): picked mon.noname-c con 0x7f9a280a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:13:21.386 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.383+0000 7f9a2f614640 10 monclient(hunting): picked mon.noname-b con 0x7f9a2813f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:13:21.386 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.383+0000 7f9a2f614640 10 monclient(hunting): picked mon.noname-a con 0x7f9a281430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:13:21.386 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.383+0000 7f9a2f614640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a280abd30 con 0x7f9a281430c0 2024-03-25T18:13:21.386 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.383+0000 7f9a2f614640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a280abfa0 con 0x7f9a2813f710 2024-03-25T18:13:21.386 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.383+0000 7f9a2f614640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a281d77a0 con 0x7f9a280a98f0 2024-03-25T18:13:21.386 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.383+0000 7f9a2f614640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:21.386 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.383+0000 7f9a2f614640 10 monclient(hunting): authenticate will time out at 7282.694387s 2024-03-25T18:13:21.393 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.392+0000 7f9a2dbb1640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f9a281430c0 0x7f9a281385d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:45632/0 (socket says 172.21.15.156:45632) 2024-03-25T18:13:21.393 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.392+0000 7f9a2dbb1640 1 -- 172.21.15.156:0/2424731784 learned_addr learned my addr 172.21.15.156:0/2424731784 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:13:21.394 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.393+0000 7f9a1dffb640 10 client.?.objecter ms_handle_connect 0x7f9a281430c0 2024-03-25T18:13:21.394 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.393+0000 7f9a1dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:21.399 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.398+0000 7f9a1dffb640 1 -- 172.21.15.156:0/2424731784 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1243493250 0 0) 0x7f9a280abd30 con 0x7f9a281430c0 2024-03-25T18:13:21.399 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.398+0000 7f9a1dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:21.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.398+0000 7f9a1dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:21.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.398+0000 7f9a1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:21.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.398+0000 7f9a1dffb640 10 monclient(hunting): my global_id is 4452 2024-03-25T18:13:21.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.398+0000 7f9a1dffb640 1 -- 172.21.15.156:0/2424731784 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a0c004eb0 con 0x7f9a281430c0 2024-03-25T18:13:21.401 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.400+0000 7f9a1dffb640 10 client.?.objecter ms_handle_connect 0x7f9a2813f710 2024-03-25T18:13:21.401 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.400+0000 7f9a1dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:21.401 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.400+0000 7f9a1dffb640 10 client.?.objecter ms_handle_connect 0x7f9a280a98f0 2024-03-25T18:13:21.402 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.400+0000 7f9a1dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:21.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.407+0000 7f9a1dffb640 1 -- 172.21.15.156:0/2424731784 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2515973099 0 0) 0x7f9a281d77a0 con 0x7f9a280a98f0 2024-03-25T18:13:21.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.407+0000 7f9a1dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:21.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.407+0000 7f9a1dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:21.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.407+0000 7f9a1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:21.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.407+0000 7f9a1dffb640 10 monclient(hunting): my global_id is 4457 2024-03-25T18:13:21.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.407+0000 7f9a1dffb640 1 -- 172.21.15.156:0/2424731784 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a280abd30 con 0x7f9a280a98f0 2024-03-25T18:13:21.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.408+0000 7f9a1dffb640 1 -- 172.21.15.156:0/2424731784 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512205776 0 0) 0x7f9a280abfa0 con 0x7f9a2813f710 2024-03-25T18:13:21.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.408+0000 7f9a1dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:21.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.408+0000 7f9a1dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:21.410 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.408+0000 7f9a1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:21.410 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.408+0000 7f9a1dffb640 10 monclient(hunting): my global_id is 4453 2024-03-25T18:13:21.410 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.408+0000 7f9a1dffb640 1 -- 172.21.15.156:0/2424731784 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a281d77a0 con 0x7f9a2813f710 2024-03-25T18:13:21.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.409+0000 7f9a1dffb640 1 -- 172.21.15.156:0/2424731784 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1299947586 0 0) 0x7f9a0c004eb0 con 0x7f9a281430c0 2024-03-25T18:13:21.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.409+0000 7f9a1dffb640 1 -- 172.21.15.156:0/2424731784 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a280abfa0 con 0x7f9a281430c0 2024-03-25T18:13:21.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.409+0000 7f9a1dffb640 1 -- 172.21.15.156:0/2424731784 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9a20001c90 con 0x7f9a281430c0 2024-03-25T18:13:21.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.409+0000 7f9a1dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:21.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.409+0000 7f9a1dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:21.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.409+0000 7f9a1dffb640 10 monclient(hunting): dump: 2024-03-25T18:13:21.411 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:21.411 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:21.411 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:21.411 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:21.411 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:21.411 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:21.412 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:21.412 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:21.412 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:21.412 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:21.412 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.409+0000 7f9a1dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:21.418 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.417+0000 7f9a1dffb640 1 -- 172.21.15.156:0/2424731784 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1500195576 0 0) 0x7f9a280abd30 con 0x7f9a280a98f0 2024-03-25T18:13:21.418 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.417+0000 7f9a1dffb640 1 -- 172.21.15.156:0/2424731784 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a0c004eb0 con 0x7f9a280a98f0 2024-03-25T18:13:21.418 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.417+0000 7f9a1dffb640 1 -- 172.21.15.156:0/2424731784 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9a24001b00 con 0x7f9a280a98f0 2024-03-25T18:13:21.418 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.417+0000 7f9a1dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:21.419 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.417+0000 7f9a1dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:21.419 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.417+0000 7f9a1dffb640 10 monclient(hunting): dump: 2024-03-25T18:13:21.419 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:21.419 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:21.419 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:21.419 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:21.419 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:21.419 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:21.419 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:21.419 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:21.419 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:21.419 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:21.420 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.419+0000 7f9a1dffb640 1 -- 172.21.15.156:0/2424731784 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 931990124 0 0) 0x7f9a281d77a0 con 0x7f9a2813f710 2024-03-25T18:13:21.421 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.419+0000 7f9a1dffb640 1 -- 172.21.15.156:0/2424731784 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a280abd30 con 0x7f9a2813f710 2024-03-25T18:13:21.421 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.419+0000 7f9a1dffb640 1 -- 172.21.15.156:0/2424731784 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9a18001830 con 0x7f9a2813f710 2024-03-25T18:13:21.421 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.419+0000 7f9a1dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:21.421 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.419+0000 7f9a1dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:21.421 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.419+0000 7f9a1dffb640 10 monclient(hunting): dump: 2024-03-25T18:13:21.421 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:21.421 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:21.421 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:21.421 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:21.421 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:21.421 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:21.422 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:21.422 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:21.422 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:21.422 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:21.423 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.422+0000 7f9a1dffb640 1 -- 172.21.15.156:0/2424731784 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3370628885 0 0) 0x7f9a280abfa0 con 0x7f9a281430c0 2024-03-25T18:13:21.423 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.422+0000 7f9a1dffb640 1 -- 172.21.15.156:0/2424731784 >> v1:172.21.15.156:6790/0 conn(0x7f9a280a98f0 legacy=0x7f9a281d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:21.423 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.422+0000 7f9a1dffb640 1 -- 172.21.15.156:0/2424731784 >> v1:172.21.15.194:6789/0 conn(0x7f9a2813f710 legacy=0x7f9a281d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:21.424 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.422+0000 7f9a1dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:13:21.424 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.422+0000 7f9a1dffb640 1 monclient: found mon.a 2024-03-25T18:13:21.424 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.422+0000 7f9a1dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:21.424 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.422+0000 7f9a1dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:13:21.424 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.422+0000 7f9a1dffb640 1 -- 172.21.15.156:0/2424731784 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a281d83d0 con 0x7f9a281430c0 2024-03-25T18:13:21.424 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.422+0000 7f9a1dffb640 10 monclient: _finish_auth 0 2024-03-25T18:13:21.424 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.422+0000 7f9a1dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:13:21.424 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.422+0000 7f9a1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:21.424 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.422+0000 7f9a2f614640 5 monclient: authenticate success, global_id 4452 2024-03-25T18:13:21.424 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.422+0000 7f9a2f614640 10 monclient: _renew_subs 2024-03-25T18:13:21.424 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.422+0000 7f9a2f614640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:13:21.424 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.422+0000 7f9a2f614640 1 -- 172.21.15.156:0/2424731784 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a281d41e0 con 0x7f9a281430c0 2024-03-25T18:13:21.425 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.422+0000 7f9a2f614640 10 client.4452.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:21.425 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.422+0000 7f9a2f614640 10 monclient: _renew_subs 2024-03-25T18:13:21.425 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.422+0000 7f9a2f614640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:13:21.425 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.422+0000 7f9a2f614640 1 -- 172.21.15.156:0/2424731784 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a281d5110 con 0x7f9a281430c0 2024-03-25T18:13:21.425 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.422+0000 7f9a2f614640 1 librados: init done 2024-03-25T18:13:21.425 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.423+0000 7f9a2f614640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:21.426 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.423+0000 7f9a2f614640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:21.426 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.423+0000 7f9a2f614640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:13:21.426 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.423+0000 7f9a2f614640 1 -- 172.21.15.156:0/2424731784 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a281d7930 con 0x7f9a281430c0 2024-03-25T18:13:21.731 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.730+0000 7f9a1dffb640 1 -- 172.21.15.156:0/2424731784 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a20003970 con 0x7f9a281430c0 2024-03-25T18:13:21.731 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.730+0000 7f9a1dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:21.731 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.730+0000 7f9a1dffb640 1 -- 172.21.15.156:0/2424731784 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9a20003d80 con 0x7f9a281430c0 2024-03-25T18:13:21.731 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.730+0000 7f9a1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:21.731 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.730+0000 7f9a1dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:21.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.730+0000 7f9a1dffb640 10 monclient: dump: 2024-03-25T18:13:21.732 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:21.732 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:21.732 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:21.732 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:21.732 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:21.732 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:21.732 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:21.732 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:21.732 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:21.732 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:21.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.730+0000 7f9a1f7fe640 4 set_mon_vals no callback set 2024-03-25T18:13:21.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.731+0000 7f9a1dffb640 1 -- 172.21.15.156:0/2424731784 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f9a20003fa0 con 0x7f9a281430c0 2024-03-25T18:13:21.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.732+0000 7f9a2d3b0640 10 monclient: get_auth_request con 0x7f9a0c06e0f0 auth_method 0 2024-03-25T18:13:21.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.732+0000 7f9a1dffb640 10 client.4452.objecter ms_handle_connect 0x7f9a0c06e0f0 2024-03-25T18:13:21.739 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.738+0000 7f9a1dffb640 1 -- 172.21.15.156:0/2424731784 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f9a20054ad0 con 0x7f9a281430c0 2024-03-25T18:13:21.739 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.738+0000 7f9a1dffb640 10 client.4452.objecter ms_dispatch 0x7f9a28000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:21.739 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.738+0000 7f9a1dffb640 3 client.4452.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:21.739 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.738+0000 7f9a1dffb640 3 client.4452.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:21.739 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.738+0000 7f9a1dffb640 20 client.4452.objecter dump_active .. 0 homeless 2024-03-25T18:13:21.739 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.738+0000 7f9a1dffb640 1 -- 172.21.15.156:0/2424731784 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9a200505e0 con 0x7f9a281430c0 2024-03-25T18:13:21.739 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.738+0000 7f9a1dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:21.739 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.738+0000 7f9a1dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:21.991 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.990+0000 7f9a2f614640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:21.991 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.990+0000 7f9a2f614640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:21.991 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.990+0000 7f9a2f614640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:13:21.991 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:21.990+0000 7f9a2f614640 1 -- 172.21.15.156:0/2424731784 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a28136a20 con 0x7f9a281430c0 2024-03-25T18:13:22.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:22.022+0000 7f9a1dffb640 1 -- 172.21.15.156:0/2424731784 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f9a20052510 con 0x7f9a281430c0 2024-03-25T18:13:22.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:22.022+0000 7f9a1dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:22.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:22.022+0000 7f9a1dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:22.023 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:13:22.024 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:22.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:22.024+0000 7f9a2f614640 10 librados: watch_flush enter 2024-03-25T18:13:22.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:22.025+0000 7f9a2f614640 10 librados: watch_flush exit 2024-03-25T18:13:22.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:22.025+0000 7f9a2f614640 20 client.4452.objecter shutdown clearing up homeless session... 2024-03-25T18:13:22.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:22.025+0000 7f9a2f614640 10 client.4452.objecter successfully canceled tick 2024-03-25T18:13:22.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:22.025+0000 7f9a2f614640 1 -- 172.21.15.156:0/2424731784 >> v1:172.21.15.194:6800/2946427713 conn(0x7f9a0c06e0f0 legacy=0x7f9a0c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:22.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:22.025+0000 7f9a2f614640 10 monclient: shutdown 2024-03-25T18:13:22.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:22.025+0000 7f9a2f614640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:22.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:22.025+0000 7f9a2f614640 1 -- 172.21.15.156:0/2424731784 >> v1:172.21.15.156:6789/0 conn(0x7f9a281430c0 legacy=0x7f9a281385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:22.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:22.025+0000 7f9a2f614640 1 -- 172.21.15.156:0/2424731784 shutdown_connections 2024-03-25T18:13:22.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:22.025+0000 7f9a2f614640 1 -- 172.21.15.156:0/2424731784 >> 172.21.15.156:0/2424731784 conn(0x7f9a280b7ea0 msgr2=0x7f9a2813d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:22.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:22.025+0000 7f9a2f614640 1 -- 172.21.15.156:0/2424731784 shutdown_connections 2024-03-25T18:13:22.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:22.025+0000 7f9a2f614640 1 -- 172.21.15.156:0/2424731784 wait complete. 2024-03-25T18:13:22.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:22.025+0000 7f9a2f614640 1 librados: shutdown 2024-03-25T18:13:22.038 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:24.005 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:05:29.996 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:24.060 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:05:30.051 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:24.100 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:05:30.091 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:24.111 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:05:30.102 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:24.117 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:05:30.084 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:24.128 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:05:30.119 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:24.133 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:05:30.100 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:24.166 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:05:30.133 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:24.180 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:05:30.171 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:28.845 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:28.959 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.958+0000 7fd4b26e0640 10 monclient: get_monmap_and_config 2024-03-25T18:13:28.959 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.958+0000 7fd4b26e0640 10 monclient: build_initial_monmap 2024-03-25T18:13:28.959 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.958+0000 7fd4b26e0640 10 monclient: monmap: 2024-03-25T18:13:28.959 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:13:28.959 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:28.960 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:13:28.959592+0000 2024-03-25T18:13:28.960 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:13:28.959592+0000 2024-03-25T18:13:28.960 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:28.960 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:28.960 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:13:28.960 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:13:28.960 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:13:28.960 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:28.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.959+0000 7fd4b26e0640 1 Processor -- start 2024-03-25T18:13:28.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.959+0000 7fd4b26e0640 1 -- start start 2024-03-25T18:13:28.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.959+0000 7fd4b26e0640 10 monclient: init 2024-03-25T18:13:28.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.960+0000 7fd4b26e0640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:28.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.960+0000 7fd4b26e0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:28.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.960+0000 7fd4b26e0640 10 monclient(hunting): picked mon.noname-a con 0x7fd4ac0b4d60 addr v1:172.21.15.156:6789/0 2024-03-25T18:13:28.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.960+0000 7fd4b26e0640 10 monclient(hunting): picked mon.noname-c con 0x7fd4ac0b1730 addr v1:172.21.15.194:6789/0 2024-03-25T18:13:28.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.960+0000 7fd4b26e0640 10 monclient(hunting): picked mon.noname-b con 0x7fd4ac14be90 addr v1:172.21.15.156:6790/0 2024-03-25T18:13:28.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.960+0000 7fd4b26e0640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4ac0af940 con 0x7fd4ac0b4d60 2024-03-25T18:13:28.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.960+0000 7fd4b26e0640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4ac0b0a70 con 0x7fd4ac0b1730 2024-03-25T18:13:28.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.960+0000 7fd4b26e0640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4ac0b0d50 con 0x7fd4ac14be90 2024-03-25T18:13:28.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.960+0000 7fd4b26e0640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:28.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.960+0000 7fd4b26e0640 10 monclient(hunting): authenticate will time out at 7290.270987s 2024-03-25T18:13:28.964 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.963+0000 7fd4abfff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fd4ac0b4d60 0x7fd4ac13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:48128/0 (socket says 172.21.15.156:48128) 2024-03-25T18:13:28.964 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.963+0000 7fd4abfff640 1 -- 172.21.15.156:0/3849977311 learned_addr learned my addr 172.21.15.156:0/3849977311 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:13:28.981 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.980+0000 7fd4aaffd640 1 -- 172.21.15.156:0/3849977311 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2953379147 0 0) 0x7fd4ac0af940 con 0x7fd4ac0b4d60 2024-03-25T18:13:28.981 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.980+0000 7fd4aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:28.981 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.980+0000 7fd4aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:28.981 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.980+0000 7fd4aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:28.981 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.980+0000 7fd4aaffd640 10 monclient(hunting): my global_id is 4455 2024-03-25T18:13:28.981 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.980+0000 7fd4aaffd640 1 -- 172.21.15.156:0/3849977311 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd494003520 con 0x7fd4ac0b4d60 2024-03-25T18:13:28.984 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.983+0000 7fd4aaffd640 1 -- 172.21.15.156:0/3849977311 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 477513818 0 0) 0x7fd4ac0b0a70 con 0x7fd4ac0b1730 2024-03-25T18:13:28.984 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.983+0000 7fd4aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:28.984 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.983+0000 7fd4aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:28.984 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.983+0000 7fd4aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:28.984 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.983+0000 7fd4aaffd640 10 monclient(hunting): my global_id is 4456 2024-03-25T18:13:28.984 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.983+0000 7fd4aaffd640 1 -- 172.21.15.156:0/3849977311 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4ac0af940 con 0x7fd4ac0b1730 2024-03-25T18:13:28.986 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.985+0000 7fd4aaffd640 1 -- 172.21.15.156:0/3849977311 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199106874 0 0) 0x7fd4ac0b0d50 con 0x7fd4ac14be90 2024-03-25T18:13:28.986 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.985+0000 7fd4aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:28.986 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.985+0000 7fd4aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:28.986 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.985+0000 7fd4aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:28.986 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.985+0000 7fd4aaffd640 10 monclient(hunting): my global_id is 4460 2024-03-25T18:13:28.986 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.985+0000 7fd4aaffd640 1 -- 172.21.15.156:0/3849977311 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4ac0b0a70 con 0x7fd4ac14be90 2024-03-25T18:13:28.997 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.995+0000 7fd4aaffd640 1 -- 172.21.15.156:0/3849977311 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 173357124 0 0) 0x7fd4ac0af940 con 0x7fd4ac0b1730 2024-03-25T18:13:28.997 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.995+0000 7fd4aaffd640 1 -- 172.21.15.156:0/3849977311 >> v1:172.21.15.156:6790/0 conn(0x7fd4ac14be90 legacy=0x7fd4ac14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:28.997 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.995+0000 7fd4aaffd640 1 -- 172.21.15.156:0/3849977311 >> v1:172.21.15.156:6789/0 conn(0x7fd4ac0b4d60 legacy=0x7fd4ac13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:28.997 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.995+0000 7fd4aaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:13:28.997 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.995+0000 7fd4aaffd640 1 monclient: found mon.noname-c 2024-03-25T18:13:28.997 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.995+0000 7fd4aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:28.997 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.996+0000 7fd4aaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:13:28.997 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.996+0000 7fd4aaffd640 1 -- 172.21.15.156:0/3849977311 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac0b1030 con 0x7fd4ac0b1730 2024-03-25T18:13:28.997 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.996+0000 7fd4aaffd640 10 monclient: _finish_auth 0 2024-03-25T18:13:28.997 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.996+0000 7fd4aaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:13:28.997 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.996+0000 7fd4aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:28.998 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.996+0000 7fd4aaffd640 1 -- 172.21.15.156:0/3849977311 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fd4a0001670 con 0x7fd4ac0b1730 2024-03-25T18:13:28.998 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.996+0000 7fd4aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:28.998 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.996+0000 7fd4aaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:28.998 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.996+0000 7fd4aaffd640 10 monclient: dump: 2024-03-25T18:13:28.998 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:28.998 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:28.998 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:28.998 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:28.998 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:28.998 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:28.998 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:28.998 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:28.998 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:28.998 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:28.998 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.996+0000 7fd4b26e0640 5 monclient: authenticate success, global_id 4456 2024-03-25T18:13:28.999 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:28.996+0000 7fd4b26e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:29.011 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.010+0000 7fd4aaffd640 1 -- 172.21.15.156:0/3849977311 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4a00017d0 con 0x7fd4ac0b1730 2024-03-25T18:13:29.011 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.010+0000 7fd4aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:29.012 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.010+0000 7fd4aaffd640 1 -- 172.21.15.156:0/3849977311 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fd4a0001be0 con 0x7fd4ac0b1730 2024-03-25T18:13:29.012 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.010+0000 7fd4b26e0640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:29.012 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.010+0000 7fd4b26e0640 4 set_mon_vals no callback set 2024-03-25T18:13:29.012 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.010+0000 7fd4aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:29.012 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.010+0000 7fd4aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:29.012 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.010+0000 7fd4aaffd640 10 monclient: dump: 2024-03-25T18:13:29.012 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:29.012 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:29.012 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:29.012 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:29.012 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:29.012 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:29.012 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:29.012 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:29.013 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:29.013 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:29.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.011+0000 7fd4b26e0640 10 monclient: shutdown 2024-03-25T18:13:29.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.011+0000 7fd4b26e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:29.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.011+0000 7fd4b26e0640 1 -- 172.21.15.156:0/3849977311 >> v1:172.21.15.194:6789/0 conn(0x7fd4ac0b1730 legacy=0x7fd4ac0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:29.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.011+0000 7fd4b26e0640 1 -- 172.21.15.156:0/3849977311 shutdown_connections 2024-03-25T18:13:29.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.011+0000 7fd4b26e0640 1 -- 172.21.15.156:0/3849977311 >> 172.21.15.156:0/3849977311 conn(0x7fd4ac0617a0 msgr2=0x7fd4ac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:29.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.011+0000 7fd4b26e0640 1 -- 172.21.15.156:0/3849977311 shutdown_connections 2024-03-25T18:13:29.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.011+0000 7fd4b26e0640 1 -- 172.21.15.156:0/3849977311 wait complete. 2024-03-25T18:13:29.015 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.012+0000 7fd4b26e0640 10 monclient: build_initial_monmap 2024-03-25T18:13:29.015 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.012+0000 7fd4b26e0640 10 monclient: monmap: 2024-03-25T18:13:29.015 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:13:29.015 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:29.015 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:13:29.015 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:13:29.015 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:29.015 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:13:29.015 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:13:29.015 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:13:29.015 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:13:29.015 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:29.015 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.013+0000 7fd4b26e0640 1 librados: starting msgr at 2024-03-25T18:13:29.015 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.013+0000 7fd4b26e0640 1 librados: starting objecter 2024-03-25T18:13:29.016 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.013+0000 7fd4b26e0640 1 Processor -- start 2024-03-25T18:13:29.016 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.013+0000 7fd4b26e0640 1 -- start start 2024-03-25T18:13:29.016 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.013+0000 7fd4b26e0640 1 librados: setting wanted keys 2024-03-25T18:13:29.016 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.013+0000 7fd4b26e0640 1 librados: calling monclient init 2024-03-25T18:13:29.016 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.013+0000 7fd4b26e0640 10 monclient: init 2024-03-25T18:13:29.017 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.014+0000 7fd4b26e0640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:29.017 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.014+0000 7fd4b26e0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:29.017 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.014+0000 7fd4b26e0640 10 monclient(hunting): picked mon.noname-c con 0x7fd4ac0b1730 addr v1:172.21.15.156:6790/0 2024-03-25T18:13:29.017 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.014+0000 7fd4b26e0640 10 monclient(hunting): picked mon.noname-b con 0x7fd4ac0b4d60 addr v1:172.21.15.194:6789/0 2024-03-25T18:13:29.017 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.014+0000 7fd4b26e0640 10 monclient(hunting): picked mon.noname-a con 0x7fd4ac14be90 addr v1:172.21.15.156:6789/0 2024-03-25T18:13:29.017 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.014+0000 7fd4b26e0640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4ac142750 con 0x7fd4ac14be90 2024-03-25T18:13:29.017 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.014+0000 7fd4b26e0640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4ac141380 con 0x7fd4ac0b4d60 2024-03-25T18:13:29.017 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.014+0000 7fd4b26e0640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4ac142470 con 0x7fd4ac0b1730 2024-03-25T18:13:29.017 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.014+0000 7fd4b26e0640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:29.017 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.014+0000 7fd4b26e0640 10 monclient(hunting): authenticate will time out at 7290.325267s 2024-03-25T18:13:29.017 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.015+0000 7fd4ab7fe640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fd4ac0b4d60 0x7fd4ac1de050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:53842/0 (socket says 172.21.15.156:53842) 2024-03-25T18:13:29.018 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.015+0000 7fd4ab7fe640 1 -- 172.21.15.156:0/451109216 learned_addr learned my addr 172.21.15.156:0/451109216 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:13:29.020 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.019+0000 7fd4a8ff9640 10 client.?.objecter ms_handle_connect 0x7fd4ac0b4d60 2024-03-25T18:13:29.020 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.019+0000 7fd4a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:29.031 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.030+0000 7fd4a8ff9640 1 -- 172.21.15.156:0/451109216 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4271804683 0 0) 0x7fd4ac141380 con 0x7fd4ac0b4d60 2024-03-25T18:13:29.031 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.030+0000 7fd4a8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:13:29.031 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.030+0000 7fd4a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:29.031 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.030+0000 7fd4a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:29.031 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.030+0000 7fd4a8ff9640 10 monclient(hunting): my global_id is 4459 2024-03-25T18:13:29.031 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.030+0000 7fd4a8ff9640 1 -- 172.21.15.156:0/451109216 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd498004eb0 con 0x7fd4ac0b4d60 2024-03-25T18:13:29.035 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.034+0000 7fd4a8ff9640 10 client.?.objecter ms_handle_connect 0x7fd4ac0b1730 2024-03-25T18:13:29.035 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.034+0000 7fd4a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:29.035 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.034+0000 7fd4a8ff9640 10 client.?.objecter ms_handle_connect 0x7fd4ac14be90 2024-03-25T18:13:29.035 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.034+0000 7fd4a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:29.041 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.040+0000 7fd4a8ff9640 1 -- 172.21.15.156:0/451109216 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3198065796 0 0) 0x7fd4ac142750 con 0x7fd4ac14be90 2024-03-25T18:13:29.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.040+0000 7fd4a8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:13:29.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.040+0000 7fd4a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:29.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.040+0000 7fd4a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:29.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.040+0000 7fd4a8ff9640 10 monclient(hunting): my global_id is 4458 2024-03-25T18:13:29.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.040+0000 7fd4a8ff9640 1 -- 172.21.15.156:0/451109216 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4ac141380 con 0x7fd4ac14be90 2024-03-25T18:13:29.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.042+0000 7fd4a8ff9640 1 -- 172.21.15.156:0/451109216 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 65100659 0 0) 0x7fd498004eb0 con 0x7fd4ac0b4d60 2024-03-25T18:13:29.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.042+0000 7fd4a8ff9640 1 -- 172.21.15.156:0/451109216 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4ac142750 con 0x7fd4ac0b4d60 2024-03-25T18:13:29.044 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.042+0000 7fd4a8ff9640 1 -- 172.21.15.156:0/451109216 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fd4a0001670 con 0x7fd4ac0b4d60 2024-03-25T18:13:29.044 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.042+0000 7fd4a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:29.044 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.042+0000 7fd4a8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:29.044 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.042+0000 7fd4a8ff9640 10 monclient(hunting): dump: 2024-03-25T18:13:29.044 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:29.044 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:29.044 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:29.044 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:29.044 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:29.044 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:29.044 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:29.044 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:29.044 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:29.044 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:29.044 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.042+0000 7fd4a8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:29.045 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.043+0000 7fd4a8ff9640 1 -- 172.21.15.156:0/451109216 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039366439 0 0) 0x7fd4ac142470 con 0x7fd4ac0b1730 2024-03-25T18:13:29.045 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.043+0000 7fd4a8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:13:29.045 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.043+0000 7fd4a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:29.045 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.043+0000 7fd4a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:29.045 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.043+0000 7fd4a8ff9640 10 monclient(hunting): my global_id is 4463 2024-03-25T18:13:29.045 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.043+0000 7fd4a8ff9640 1 -- 172.21.15.156:0/451109216 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd498004eb0 con 0x7fd4ac0b1730 2024-03-25T18:13:29.054 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.053+0000 7fd4a8ff9640 1 -- 172.21.15.156:0/451109216 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2838298745 0 0) 0x7fd4ac142750 con 0x7fd4ac0b4d60 2024-03-25T18:13:29.054 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.053+0000 7fd4a8ff9640 1 -- 172.21.15.156:0/451109216 >> v1:172.21.15.156:6790/0 conn(0x7fd4ac0b1730 legacy=0x7fd4ac140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:29.054 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.053+0000 7fd4a8ff9640 1 -- 172.21.15.156:0/451109216 >> v1:172.21.15.156:6789/0 conn(0x7fd4ac14be90 legacy=0x7fd4ac1de760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:29.054 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.053+0000 7fd4a8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:13:29.054 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.053+0000 7fd4a8ff9640 1 monclient: found mon.b 2024-03-25T18:13:29.055 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.053+0000 7fd4a8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:29.055 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.053+0000 7fd4a8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:13:29.055 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.053+0000 7fd4a8ff9640 1 -- 172.21.15.156:0/451109216 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac1dee70 con 0x7fd4ac0b4d60 2024-03-25T18:13:29.055 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.053+0000 7fd4a8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:13:29.055 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.053+0000 7fd4a8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:13:29.055 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.053+0000 7fd4a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:29.055 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.053+0000 7fd4b26e0640 5 monclient: authenticate success, global_id 4459 2024-03-25T18:13:29.055 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.053+0000 7fd4b26e0640 10 monclient: _renew_subs 2024-03-25T18:13:29.055 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.053+0000 7fd4b26e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:13:29.055 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.053+0000 7fd4b26e0640 1 -- 172.21.15.156:0/451109216 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4ac139db0 con 0x7fd4ac0b4d60 2024-03-25T18:13:29.055 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.053+0000 7fd4b26e0640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:29.055 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.053+0000 7fd4b26e0640 10 monclient: _renew_subs 2024-03-25T18:13:29.055 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.053+0000 7fd4b26e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:13:29.055 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.053+0000 7fd4b26e0640 1 -- 172.21.15.156:0/451109216 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4ac143160 con 0x7fd4ac0b4d60 2024-03-25T18:13:29.056 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.053+0000 7fd4b26e0640 1 librados: init done 2024-03-25T18:13:29.056 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.054+0000 7fd4b26e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:29.056 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.054+0000 7fd4b26e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:29.056 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.054+0000 7fd4b26e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:13:29.056 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.054+0000 7fd4b26e0640 1 -- 172.21.15.156:0/451109216 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4ac143460 con 0x7fd4ac0b4d60 2024-03-25T18:13:29.339 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.338+0000 7fd4a8ff9640 1 -- 172.21.15.156:0/451109216 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4a00038a0 con 0x7fd4ac0b4d60 2024-03-25T18:13:29.339 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.338+0000 7fd4a8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:29.339 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.338+0000 7fd4a8ff9640 1 -- 172.21.15.156:0/451109216 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fd4a0003a90 con 0x7fd4ac0b4d60 2024-03-25T18:13:29.340 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.338+0000 7fd4a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:29.340 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.338+0000 7fd4a8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:29.340 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.338+0000 7fd4a8ff9640 10 monclient: dump: 2024-03-25T18:13:29.340 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:29.340 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:29.340 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:29.340 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:29.340 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:29.340 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:29.340 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:29.340 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:29.340 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:29.341 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:29.341 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.338+0000 7fd4aa7fc640 4 set_mon_vals no callback set 2024-03-25T18:13:29.341 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.339+0000 7fd4a8ff9640 1 -- 172.21.15.156:0/451109216 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7fd4a0003cb0 con 0x7fd4ac0b4d60 2024-03-25T18:13:29.341 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.339+0000 7fd4abfff640 10 monclient: get_auth_request con 0x7fd49806de00 auth_method 0 2024-03-25T18:13:29.341 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.340+0000 7fd4a8ff9640 10 client.4459.objecter ms_handle_connect 0x7fd49806de00 2024-03-25T18:13:29.346 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.345+0000 7fd4a8ff9640 1 -- 172.21.15.156:0/451109216 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7fd4a0054760 con 0x7fd4ac0b4d60 2024-03-25T18:13:29.346 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.345+0000 7fd4a8ff9640 10 client.4459.objecter ms_dispatch 0x7fd4ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:29.346 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.345+0000 7fd4a8ff9640 3 client.4459.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:29.346 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.345+0000 7fd4a8ff9640 3 client.4459.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:29.346 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.345+0000 7fd4a8ff9640 20 client.4459.objecter dump_active .. 0 homeless 2024-03-25T18:13:29.347 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.346+0000 7fd4a8ff9640 1 -- 172.21.15.156:0/451109216 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd4a0050270 con 0x7fd4ac0b4d60 2024-03-25T18:13:29.347 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.346+0000 7fd4a8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:29.347 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.346+0000 7fd4a8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:29.595 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.594+0000 7fd4b26e0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:29.596 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.594+0000 7fd4b26e0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:29.596 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.594+0000 7fd4b26e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:13:29.596 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.594+0000 7fd4b26e0640 1 -- 172.21.15.156:0/451109216 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4ac142e80 con 0x7fd4ac0b4d60 2024-03-25T18:13:29.633 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.632+0000 7fd4a8ff9640 1 -- 172.21.15.156:0/451109216 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7fd4ac142e80 con 0x7fd4ac0b4d60 2024-03-25T18:13:29.633 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.632+0000 7fd4a8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:29.634 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.632+0000 7fd4a8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:29.634 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:13:29.634 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:29.636 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.635+0000 7fd4b26e0640 10 librados: watch_flush enter 2024-03-25T18:13:29.636 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.635+0000 7fd4b26e0640 10 librados: watch_flush exit 2024-03-25T18:13:29.637 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.635+0000 7fd4b26e0640 20 client.4459.objecter shutdown clearing up homeless session... 2024-03-25T18:13:29.637 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.635+0000 7fd4b26e0640 10 client.4459.objecter successfully canceled tick 2024-03-25T18:13:29.637 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.635+0000 7fd4b26e0640 1 -- 172.21.15.156:0/451109216 >> v1:172.21.15.194:6800/2946427713 conn(0x7fd49806de00 legacy=0x7fd4980702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:29.637 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.635+0000 7fd4b26e0640 10 monclient: shutdown 2024-03-25T18:13:29.637 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.635+0000 7fd4b26e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:29.637 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.635+0000 7fd4b26e0640 1 -- 172.21.15.156:0/451109216 >> v1:172.21.15.194:6789/0 conn(0x7fd4ac0b4d60 legacy=0x7fd4ac1de050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:29.637 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.635+0000 7fd4b26e0640 1 -- 172.21.15.156:0/451109216 shutdown_connections 2024-03-25T18:13:29.637 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.635+0000 7fd4b26e0640 1 -- 172.21.15.156:0/451109216 >> 172.21.15.156:0/451109216 conn(0x7fd4ac0617a0 msgr2=0x7fd4ac13f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:29.637 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.635+0000 7fd4b26e0640 1 -- 172.21.15.156:0/451109216 shutdown_connections 2024-03-25T18:13:29.637 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.636+0000 7fd4b26e0640 1 -- 172.21.15.156:0/451109216 wait complete. 2024-03-25T18:13:29.637 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:29.636+0000 7fd4b26e0640 1 librados: shutdown 2024-03-25T18:13:29.649 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:36.456 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:36.569 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.568+0000 7f7bb5657640 10 monclient: get_monmap_and_config 2024-03-25T18:13:36.569 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.568+0000 7f7bb5657640 10 monclient: build_initial_monmap 2024-03-25T18:13:36.570 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.568+0000 7f7bb5657640 10 monclient: monmap: 2024-03-25T18:13:36.570 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:13:36.570 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:36.570 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:13:36.569645+0000 2024-03-25T18:13:36.570 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:13:36.569645+0000 2024-03-25T18:13:36.570 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:36.570 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:36.570 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:13:36.570 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:13:36.570 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:13:36.570 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:36.572 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.569+0000 7f7bb5657640 1 Processor -- start 2024-03-25T18:13:36.572 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.569+0000 7f7bb5657640 1 -- start start 2024-03-25T18:13:36.572 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.569+0000 7f7bb5657640 10 monclient: init 2024-03-25T18:13:36.572 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.570+0000 7f7bb5657640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:36.572 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.570+0000 7f7bb5657640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:36.572 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.570+0000 7f7bb5657640 10 monclient(hunting): picked mon.noname-a con 0x7f7bb00a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:13:36.572 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.570+0000 7f7bb5657640 10 monclient(hunting): picked mon.noname-c con 0x7f7bb013f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:13:36.572 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.570+0000 7f7bb5657640 10 monclient(hunting): picked mon.noname-b con 0x7f7bb01430c0 addr v1:172.21.15.156:6790/0 2024-03-25T18:13:36.572 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.570+0000 7f7bb5657640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb0142520 con 0x7f7bb00a98f0 2024-03-25T18:13:36.572 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.570+0000 7f7bb5657640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb00b28a0 con 0x7f7bb013f710 2024-03-25T18:13:36.572 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.570+0000 7f7bb5657640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb00b2a30 con 0x7f7bb01430c0 2024-03-25T18:13:36.572 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.570+0000 7f7bb5657640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:36.572 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.570+0000 7f7bb5657640 10 monclient(hunting): authenticate will time out at 7297.881151s 2024-03-25T18:13:36.585 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.584+0000 7f7baeffd640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f7bb00a98f0 0x7f7bb00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:55650/0 (socket says 172.21.15.156:55650) 2024-03-25T18:13:36.585 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.584+0000 7f7baeffd640 1 -- 172.21.15.156:0/1824913464 learned_addr learned my addr 172.21.15.156:0/1824913464 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:13:36.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.606+0000 7f7bae7fc640 1 -- 172.21.15.156:0/1824913464 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2265342538 0 0) 0x7f7bb0142520 con 0x7f7bb00a98f0 2024-03-25T18:13:36.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.606+0000 7f7bae7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:36.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.606+0000 7f7bae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:36.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.606+0000 7f7bae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:36.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.606+0000 7f7bae7fc640 10 monclient(hunting): my global_id is 4461 2024-03-25T18:13:36.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.606+0000 7f7bae7fc640 1 -- 172.21.15.156:0/1824913464 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b9c003520 con 0x7f7bb00a98f0 2024-03-25T18:13:36.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.607+0000 7f7bae7fc640 1 -- 172.21.15.156:0/1824913464 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2324655437 0 0) 0x7f7bb00b28a0 con 0x7f7bb013f710 2024-03-25T18:13:36.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.607+0000 7f7bae7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:36.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.607+0000 7f7bae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:36.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.607+0000 7f7bae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:36.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.607+0000 7f7bae7fc640 10 monclient(hunting): my global_id is 4462 2024-03-25T18:13:36.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.607+0000 7f7bae7fc640 1 -- 172.21.15.156:0/1824913464 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bb0142520 con 0x7f7bb013f710 2024-03-25T18:13:36.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.608+0000 7f7bae7fc640 1 -- 172.21.15.156:0/1824913464 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053940292 0 0) 0x7f7bb00b2a30 con 0x7f7bb01430c0 2024-03-25T18:13:36.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.608+0000 7f7bae7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:36.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.608+0000 7f7bae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:36.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.608+0000 7f7bae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:36.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.608+0000 7f7bae7fc640 10 monclient(hunting): my global_id is 4466 2024-03-25T18:13:36.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.608+0000 7f7bae7fc640 1 -- 172.21.15.156:0/1824913464 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bb00b28a0 con 0x7f7bb01430c0 2024-03-25T18:13:36.637 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.636+0000 7f7bae7fc640 1 -- 172.21.15.156:0/1824913464 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3503600657 0 0) 0x7f7bb00b28a0 con 0x7f7bb01430c0 2024-03-25T18:13:36.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.636+0000 7f7bae7fc640 1 -- 172.21.15.156:0/1824913464 >> v1:172.21.15.194:6789/0 conn(0x7f7bb013f710 legacy=0x7f7bb01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.636+0000 7f7bae7fc640 1 -- 172.21.15.156:0/1824913464 >> v1:172.21.15.156:6789/0 conn(0x7f7bb00a98f0 legacy=0x7f7bb00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.636+0000 7f7bae7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:13:36.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.636+0000 7f7bae7fc640 1 monclient: found mon.noname-b 2024-03-25T18:13:36.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.636+0000 7f7bae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:36.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.636+0000 7f7bae7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.156:6790/0 2024-03-25T18:13:36.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.636+0000 7f7bae7fc640 1 -- 172.21.15.156:0/1824913464 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb00b44c0 con 0x7f7bb01430c0 2024-03-25T18:13:36.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.636+0000 7f7bae7fc640 10 monclient: _finish_auth 0 2024-03-25T18:13:36.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.636+0000 7f7bae7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:13:36.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.636+0000 7f7bae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:36.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.636+0000 7f7bae7fc640 1 -- 172.21.15.156:0/1824913464 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7ba8001930 con 0x7f7bb01430c0 2024-03-25T18:13:36.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.636+0000 7f7bb5657640 5 monclient: authenticate success, global_id 4466 2024-03-25T18:13:36.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.636+0000 7f7bb5657640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:36.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.636+0000 7f7bae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:36.639 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.636+0000 7f7bae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:36.639 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.636+0000 7f7bae7fc640 10 monclient: dump: 2024-03-25T18:13:36.639 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:36.639 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:36.639 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:36.639 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:36.639 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:36.639 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:36.639 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:36.639 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:36.639 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:36.639 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:36.639 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.636+0000 7f7bb5657640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:36.660 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.659+0000 7f7bae7fc640 1 -- 172.21.15.156:0/1824913464 <== mon.2 v1:172.21.15.156:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ba8001a90 con 0x7f7bb01430c0 2024-03-25T18:13:36.660 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.659+0000 7f7bae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:36.660 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.659+0000 7f7bae7fc640 1 -- 172.21.15.156:0/1824913464 <== mon.2 v1:172.21.15.156:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7ba8001ee0 con 0x7f7bb01430c0 2024-03-25T18:13:36.660 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.659+0000 7f7bae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:36.660 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.659+0000 7f7bae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:36.660 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.659+0000 7f7bae7fc640 10 monclient: dump: 2024-03-25T18:13:36.660 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:36.660 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:36.660 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:36.660 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:36.660 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:36.660 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:36.660 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:36.660 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:36.660 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:36.661 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:36.661 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.659+0000 7f7bb5657640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:36.661 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.659+0000 7f7bb5657640 4 set_mon_vals no callback set 2024-03-25T18:13:36.662 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.659+0000 7f7bb5657640 10 monclient: shutdown 2024-03-25T18:13:36.662 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.659+0000 7f7bb5657640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:36.662 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.659+0000 7f7bb5657640 1 -- 172.21.15.156:0/1824913464 >> v1:172.21.15.156:6790/0 conn(0x7f7bb01430c0 legacy=0x7f7bb014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.662 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.660+0000 7f7bb5657640 1 -- 172.21.15.156:0/1824913464 shutdown_connections 2024-03-25T18:13:36.662 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.660+0000 7f7bb5657640 1 -- 172.21.15.156:0/1824913464 >> 172.21.15.156:0/1824913464 conn(0x7f7bb00b7ea0 msgr2=0x7f7bb00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:36.662 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.660+0000 7f7bb5657640 1 -- 172.21.15.156:0/1824913464 shutdown_connections 2024-03-25T18:13:36.662 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.660+0000 7f7bb5657640 1 -- 172.21.15.156:0/1824913464 wait complete. 2024-03-25T18:13:36.662 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.660+0000 7f7bb5657640 10 monclient: build_initial_monmap 2024-03-25T18:13:36.662 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.660+0000 7f7bb5657640 10 monclient: monmap: 2024-03-25T18:13:36.663 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:13:36.663 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:36.663 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:13:36.663 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:13:36.663 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:36.663 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:13:36.663 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:13:36.663 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:13:36.663 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:13:36.663 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:36.663 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.661+0000 7f7bb5657640 1 librados: starting msgr at 2024-03-25T18:13:36.663 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.661+0000 7f7bb5657640 1 librados: starting objecter 2024-03-25T18:13:36.663 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.661+0000 7f7bb5657640 1 Processor -- start 2024-03-25T18:13:36.663 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.661+0000 7f7bb5657640 1 -- start start 2024-03-25T18:13:36.663 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.661+0000 7f7bb5657640 1 librados: setting wanted keys 2024-03-25T18:13:36.663 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.661+0000 7f7bb5657640 1 librados: calling monclient init 2024-03-25T18:13:36.664 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.661+0000 7f7bb5657640 10 monclient: init 2024-03-25T18:13:36.665 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.662+0000 7f7bb5657640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:36.665 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.662+0000 7f7bb5657640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:13:36.665 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.662+0000 7f7bb5657640 10 monclient(hunting): picked mon.noname-a con 0x7f7bb00a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:13:36.665 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.662+0000 7f7bb5657640 10 monclient(hunting): picked mon.noname-c con 0x7f7bb013f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:13:36.665 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.662+0000 7f7bb5657640 10 monclient(hunting): picked mon.noname-b con 0x7f7bb01430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:13:36.665 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.662+0000 7f7bb5657640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb01399d0 con 0x7f7bb00a98f0 2024-03-25T18:13:36.665 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.662+0000 7f7bb5657640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb01db930 con 0x7f7bb01430c0 2024-03-25T18:13:36.665 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.662+0000 7f7bb5657640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb01daa20 con 0x7f7bb013f710 2024-03-25T18:13:36.665 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.662+0000 7f7bb5657640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:36.665 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.662+0000 7f7bb5657640 10 monclient(hunting): authenticate will time out at 7297.973631s 2024-03-25T18:13:36.666 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.665+0000 7f7baf7fe640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f7bb01430c0 0x7f7bb0133f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:37366/0 (socket says 172.21.15.156:37366) 2024-03-25T18:13:36.666 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.665+0000 7f7baf7fe640 1 -- 172.21.15.156:0/1309308629 learned_addr learned my addr 172.21.15.156:0/1309308629 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:13:36.667 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.666+0000 7f7ba7fff640 10 client.?.objecter ms_handle_connect 0x7f7bb01430c0 2024-03-25T18:13:36.667 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.666+0000 7f7ba7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:36.678 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.677+0000 7f7ba7fff640 10 client.?.objecter ms_handle_connect 0x7f7bb00a98f0 2024-03-25T18:13:36.678 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.677+0000 7f7ba7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:36.679 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.678+0000 7f7ba7fff640 1 -- 172.21.15.156:0/1309308629 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679098759 0 0) 0x7f7bb01db930 con 0x7f7bb01430c0 2024-03-25T18:13:36.679 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.678+0000 7f7ba7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:36.679 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.678+0000 7f7ba7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:36.679 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.678+0000 7f7ba7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:36.679 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.678+0000 7f7ba7fff640 10 monclient(hunting): my global_id is 4465 2024-03-25T18:13:36.679 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.678+0000 7f7ba7fff640 1 -- 172.21.15.156:0/1309308629 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b90004eb0 con 0x7f7bb01430c0 2024-03-25T18:13:36.680 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.679+0000 7f7ba7fff640 10 client.?.objecter ms_handle_connect 0x7f7bb013f710 2024-03-25T18:13:36.680 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.679+0000 7f7ba7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:36.689 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.688+0000 7f7ba7fff640 1 -- 172.21.15.156:0/1309308629 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2158389738 0 0) 0x7f7b90004eb0 con 0x7f7bb01430c0 2024-03-25T18:13:36.689 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.688+0000 7f7ba7fff640 1 -- 172.21.15.156:0/1309308629 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7bb01db930 con 0x7f7bb01430c0 2024-03-25T18:13:36.689 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.688+0000 7f7ba7fff640 1 -- 172.21.15.156:0/1309308629 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7ba8001ea0 con 0x7f7bb01430c0 2024-03-25T18:13:36.690 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.688+0000 7f7ba7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:36.690 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.688+0000 7f7ba7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:36.690 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.688+0000 7f7ba7fff640 10 monclient(hunting): dump: 2024-03-25T18:13:36.690 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:36.690 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:36.690 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:36.690 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:36.690 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:36.690 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:36.690 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:36.690 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:36.690 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:36.690 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:36.690 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.688+0000 7f7ba7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:36.691 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.690+0000 7f7ba7fff640 1 -- 172.21.15.156:0/1309308629 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3737975177 0 0) 0x7f7bb01399d0 con 0x7f7bb00a98f0 2024-03-25T18:13:36.691 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.690+0000 7f7ba7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:36.691 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.690+0000 7f7ba7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:36.691 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.690+0000 7f7ba7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:36.691 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.690+0000 7f7ba7fff640 10 monclient(hunting): my global_id is 4464 2024-03-25T18:13:36.691 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.690+0000 7f7ba7fff640 1 -- 172.21.15.156:0/1309308629 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b90004eb0 con 0x7f7bb00a98f0 2024-03-25T18:13:36.698 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.696+0000 7f7ba7fff640 1 -- 172.21.15.156:0/1309308629 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 594645560 0 0) 0x7f7bb01db930 con 0x7f7bb01430c0 2024-03-25T18:13:36.698 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.696+0000 7f7ba7fff640 1 -- 172.21.15.156:0/1309308629 >> v1:172.21.15.156:6790/0 conn(0x7f7bb013f710 legacy=0x7f7bb01db220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.698 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.696+0000 7f7ba7fff640 1 -- 172.21.15.156:0/1309308629 >> v1:172.21.15.156:6789/0 conn(0x7f7bb00a98f0 legacy=0x7f7bb0137a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.698 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.696+0000 7f7ba7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:13:36.698 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.696+0000 7f7ba7fff640 1 monclient: found mon.b 2024-03-25T18:13:36.698 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.696+0000 7f7ba7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:36.698 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.696+0000 7f7ba7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:13:36.698 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.696+0000 7f7ba7fff640 1 -- 172.21.15.156:0/1309308629 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb01dad00 con 0x7f7bb01430c0 2024-03-25T18:13:36.698 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.696+0000 7f7ba7fff640 10 monclient: _finish_auth 0 2024-03-25T18:13:36.698 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.696+0000 7f7ba7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:13:36.698 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.696+0000 7f7ba7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:36.698 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.696+0000 7f7bb5657640 5 monclient: authenticate success, global_id 4465 2024-03-25T18:13:36.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.696+0000 7f7bb5657640 10 monclient: _renew_subs 2024-03-25T18:13:36.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.696+0000 7f7bb5657640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:13:36.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.696+0000 7f7bb5657640 1 -- 172.21.15.156:0/1309308629 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb0134690 con 0x7f7bb01430c0 2024-03-25T18:13:36.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.696+0000 7f7bb5657640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:36.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.696+0000 7f7bb5657640 10 monclient: _renew_subs 2024-03-25T18:13:36.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.696+0000 7f7bb5657640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:13:36.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.696+0000 7f7bb5657640 1 -- 172.21.15.156:0/1309308629 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb0135040 con 0x7f7bb01430c0 2024-03-25T18:13:36.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.696+0000 7f7bb5657640 1 librados: init done 2024-03-25T18:13:36.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.697+0000 7f7bb5657640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:36.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.697+0000 7f7bb5657640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:36.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.697+0000 7f7bb5657640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:13:36.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.697+0000 7f7bb5657640 1 -- 172.21.15.156:0/1309308629 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb00627b0 con 0x7f7bb01430c0 2024-03-25T18:13:36.987 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.986+0000 7f7ba7fff640 1 -- 172.21.15.156:0/1309308629 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ba8003c20 con 0x7f7bb01430c0 2024-03-25T18:13:36.987 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.986+0000 7f7ba7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:36.987 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.986+0000 7f7badffb640 4 set_mon_vals no callback set 2024-03-25T18:13:36.987 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.986+0000 7f7ba7fff640 1 -- 172.21.15.156:0/1309308629 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7ba8001590 con 0x7f7bb01430c0 2024-03-25T18:13:36.987 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.986+0000 7f7ba7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:36.987 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.986+0000 7f7ba7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:36.987 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.986+0000 7f7ba7fff640 10 monclient: dump: 2024-03-25T18:13:36.987 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:36.987 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:36.988 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:36.988 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:36.988 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:36.988 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:36.988 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:36.988 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:36.988 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:36.988 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:36.989 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.987+0000 7f7ba7fff640 1 -- 172.21.15.156:0/1309308629 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f7ba8004070 con 0x7f7bb01430c0 2024-03-25T18:13:36.989 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.987+0000 7f7baeffd640 10 monclient: get_auth_request con 0x7f7b9006dd90 auth_method 0 2024-03-25T18:13:36.989 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.987+0000 7f7ba7fff640 10 client.4465.objecter ms_handle_connect 0x7f7b9006dd90 2024-03-25T18:13:36.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.993+0000 7f7ba7fff640 1 -- 172.21.15.156:0/1309308629 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f7ba8054ba0 con 0x7f7bb01430c0 2024-03-25T18:13:36.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.993+0000 7f7ba7fff640 10 client.4465.objecter ms_dispatch 0x7f7bb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:36.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.993+0000 7f7ba7fff640 3 client.4465.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:36.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.993+0000 7f7ba7fff640 3 client.4465.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:36.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.993+0000 7f7ba7fff640 20 client.4465.objecter dump_active .. 0 homeless 2024-03-25T18:13:36.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.994+0000 7f7ba7fff640 1 -- 172.21.15.156:0/1309308629 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7ba80506b0 con 0x7f7bb01430c0 2024-03-25T18:13:36.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.994+0000 7f7ba7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:36.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:36.994+0000 7f7ba7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:37.232 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:37.231+0000 7f7bb5657640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:37.233 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:37.231+0000 7f7bb5657640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:37.233 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:37.231+0000 7f7bb5657640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:13:37.233 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:37.231+0000 7f7bb5657640 1 -- 172.21.15.156:0/1309308629 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7bb0141c20 con 0x7f7bb01430c0 2024-03-25T18:13:37.259 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:37.257+0000 7f7ba7fff640 1 -- 172.21.15.156:0/1309308629 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f7ba80525e0 con 0x7f7bb01430c0 2024-03-25T18:13:37.259 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:37.257+0000 7f7ba7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:37.259 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:37.257+0000 7f7ba7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:37.259 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:13:37.260 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:37.261 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:37.260+0000 7f7bb5657640 10 librados: watch_flush enter 2024-03-25T18:13:37.261 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:37.260+0000 7f7bb5657640 10 librados: watch_flush exit 2024-03-25T18:13:37.261 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:37.260+0000 7f7bb5657640 20 client.4465.objecter shutdown clearing up homeless session... 2024-03-25T18:13:37.261 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:37.260+0000 7f7bb5657640 10 client.4465.objecter successfully canceled tick 2024-03-25T18:13:37.262 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:37.260+0000 7f7bb5657640 1 -- 172.21.15.156:0/1309308629 >> v1:172.21.15.194:6800/2946427713 conn(0x7f7b9006dd90 legacy=0x7f7b90070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:37.262 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:37.260+0000 7f7bb5657640 10 monclient: shutdown 2024-03-25T18:13:37.262 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:37.260+0000 7f7bb5657640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:37.262 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:37.260+0000 7f7bb5657640 1 -- 172.21.15.156:0/1309308629 >> v1:172.21.15.194:6789/0 conn(0x7f7bb01430c0 legacy=0x7f7bb0133f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:37.262 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:37.260+0000 7f7bb5657640 1 -- 172.21.15.156:0/1309308629 shutdown_connections 2024-03-25T18:13:37.262 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:37.260+0000 7f7bb5657640 1 -- 172.21.15.156:0/1309308629 >> 172.21.15.156:0/1309308629 conn(0x7f7bb00b7ea0 msgr2=0x7f7bb0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:37.262 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:37.260+0000 7f7bb5657640 1 -- 172.21.15.156:0/1309308629 shutdown_connections 2024-03-25T18:13:37.262 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:37.260+0000 7f7bb5657640 1 -- 172.21.15.156:0/1309308629 wait complete. 2024-03-25T18:13:37.262 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:37.260+0000 7f7bb5657640 1 librados: shutdown 2024-03-25T18:13:37.272 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:44.078 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:44.092 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:05:50.083 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:44.101 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:05:50.092 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:44.130 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:05:50.120 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:44.155 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:05:50.122 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:44.160 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:05:50.127 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:44.165 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:05:50.156 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:44.170 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:05:50.160 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:44.174 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.173+0000 7f464f072640 10 monclient: get_monmap_and_config 2024-03-25T18:13:44.174 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.173+0000 7f464f072640 10 monclient: build_initial_monmap 2024-03-25T18:13:44.175 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.173+0000 7f464f072640 10 monclient: monmap: 2024-03-25T18:13:44.175 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:13:44.175 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:44.175 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:13:44.174673+0000 2024-03-25T18:13:44.175 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:13:44.174673+0000 2024-03-25T18:13:44.175 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:44.175 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:44.175 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:13:44.175 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:13:44.175 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:13:44.175 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:44.176 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.174+0000 7f464f072640 1 Processor -- start 2024-03-25T18:13:44.176 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.174+0000 7f464f072640 1 -- start start 2024-03-25T18:13:44.176 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.174+0000 7f464f072640 10 monclient: init 2024-03-25T18:13:44.176 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.175+0000 7f464f072640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:44.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.175+0000 7f464f072640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:44.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.175+0000 7f464f072640 10 monclient(hunting): picked mon.noname-c con 0x7f46480b4d60 addr v1:172.21.15.194:6789/0 2024-03-25T18:13:44.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.175+0000 7f464f072640 10 monclient(hunting): picked mon.noname-a con 0x7f46480b1730 addr v1:172.21.15.156:6789/0 2024-03-25T18:13:44.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.175+0000 7f464f072640 10 monclient(hunting): picked mon.noname-b con 0x7f464814be90 addr v1:172.21.15.156:6790/0 2024-03-25T18:13:44.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.175+0000 7f464f072640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46480af940 con 0x7f46480b1730 2024-03-25T18:13:44.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.175+0000 7f464f072640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46480b0a70 con 0x7f46480b4d60 2024-03-25T18:13:44.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.175+0000 7f464f072640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46480b0d50 con 0x7f464814be90 2024-03-25T18:13:44.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.175+0000 7f464f072640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:44.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.175+0000 7f464f072640 10 monclient(hunting): authenticate will time out at 7305.485974s 2024-03-25T18:13:44.179 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.178+0000 7f464d60f640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f464814be90 0x7f464814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:53636/0 (socket says 172.21.15.156:53636) 2024-03-25T18:13:44.179 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.178+0000 7f464d60f640 1 -- 172.21.15.156:0/2715997402 learned_addr learned my addr 172.21.15.156:0/2715997402 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:13:44.195 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.194+0000 7f46477fe640 1 -- 172.21.15.156:0/2715997402 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3861979269 0 0) 0x7f46480b0d50 con 0x7f464814be90 2024-03-25T18:13:44.195 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.194+0000 7f46477fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:44.195 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.194+0000 7f46477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:44.195 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.194+0000 7f46477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:44.195 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.194+0000 7f46477fe640 10 monclient(hunting): my global_id is 4472 2024-03-25T18:13:44.195 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.194+0000 7f46477fe640 1 -- 172.21.15.156:0/2715997402 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4638003520 con 0x7f464814be90 2024-03-25T18:13:44.197 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:05:50.164 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:44.207 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:05:50.197 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:44.208 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.207+0000 7f46477fe640 1 -- 172.21.15.156:0/2715997402 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2865137869 0 0) 0x7f4638003520 con 0x7f464814be90 2024-03-25T18:13:44.208 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.207+0000 7f46477fe640 1 -- 172.21.15.156:0/2715997402 >> v1:172.21.15.194:6789/0 conn(0x7f46480b4d60 legacy=0x7f464813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:44.209 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.207+0000 7f46477fe640 1 -- 172.21.15.156:0/2715997402 >> v1:172.21.15.156:6789/0 conn(0x7f46480b1730 legacy=0x7f46480b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:13:44.209 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.207+0000 7f46477fe640 10 monclient: _finish_hunting 0 2024-03-25T18:13:44.209 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.207+0000 7f46477fe640 1 monclient: found mon.noname-b 2024-03-25T18:13:44.209 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.207+0000 7f46477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:44.209 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.207+0000 7f46477fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.156:6790/0 2024-03-25T18:13:44.209 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.207+0000 7f46477fe640 1 -- 172.21.15.156:0/2715997402 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46480b1030 con 0x7f464814be90 2024-03-25T18:13:44.209 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.207+0000 7f46477fe640 10 monclient: _finish_auth 0 2024-03-25T18:13:44.209 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.207+0000 7f46477fe640 10 monclient: _check_auth_tickets 2024-03-25T18:13:44.209 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.207+0000 7f46477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:44.209 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.207+0000 7f46477fe640 1 -- 172.21.15.156:0/2715997402 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f4640001a10 con 0x7f464814be90 2024-03-25T18:13:44.209 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.207+0000 7f464f072640 5 monclient: authenticate success, global_id 4472 2024-03-25T18:13:44.209 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.207+0000 7f464f072640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:44.209 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.207+0000 7f46477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:44.210 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.207+0000 7f46477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:44.210 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.207+0000 7f46477fe640 10 monclient: dump: 2024-03-25T18:13:44.210 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:44.210 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:44.210 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:44.210 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:44.210 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:44.210 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:44.210 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:44.210 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:44.210 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:44.210 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:44.210 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.207+0000 7f464f072640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:44.262 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.261+0000 7f46477fe640 1 -- 172.21.15.156:0/2715997402 <== mon.2 v1:172.21.15.156:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4640001be0 con 0x7f464814be90 2024-03-25T18:13:44.262 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.261+0000 7f46477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:44.262 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.261+0000 7f46477fe640 1 -- 172.21.15.156:0/2715997402 <== mon.2 v1:172.21.15.156:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f4640002030 con 0x7f464814be90 2024-03-25T18:13:44.263 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.261+0000 7f46477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:44.263 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.261+0000 7f46477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:44.263 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.261+0000 7f46477fe640 10 monclient: dump: 2024-03-25T18:13:44.263 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:44.263 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:44.263 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:44.263 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:44.263 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:44.263 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:44.263 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:44.263 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:44.263 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:44.263 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:44.263 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.261+0000 7f464f072640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:44.263 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.261+0000 7f464f072640 4 set_mon_vals no callback set 2024-03-25T18:13:44.265 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.262+0000 7f464f072640 10 monclient: shutdown 2024-03-25T18:13:44.265 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.262+0000 7f464f072640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:44.265 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.262+0000 7f464f072640 1 -- 172.21.15.156:0/2715997402 >> v1:172.21.15.156:6790/0 conn(0x7f464814be90 legacy=0x7f464814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:44.265 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.262+0000 7f464f072640 1 -- 172.21.15.156:0/2715997402 shutdown_connections 2024-03-25T18:13:44.266 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.262+0000 7f464f072640 1 -- 172.21.15.156:0/2715997402 >> 172.21.15.156:0/2715997402 conn(0x7f46480617a0 msgr2=0x7f4648061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:44.266 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.262+0000 7f464f072640 1 -- 172.21.15.156:0/2715997402 shutdown_connections 2024-03-25T18:13:44.266 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.262+0000 7f464f072640 1 -- 172.21.15.156:0/2715997402 wait complete. 2024-03-25T18:13:44.266 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.263+0000 7f464f072640 10 monclient: build_initial_monmap 2024-03-25T18:13:44.266 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.263+0000 7f464f072640 10 monclient: monmap: 2024-03-25T18:13:44.266 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:13:44.266 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:44.266 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:13:44.266 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:13:44.266 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:44.266 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:13:44.266 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:13:44.266 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:13:44.267 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:13:44.267 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:44.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.263+0000 7f464f072640 1 librados: starting msgr at 2024-03-25T18:13:44.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.263+0000 7f464f072640 1 librados: starting objecter 2024-03-25T18:13:44.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.263+0000 7f464f072640 1 Processor -- start 2024-03-25T18:13:44.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.263+0000 7f464f072640 1 -- start start 2024-03-25T18:13:44.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.263+0000 7f464f072640 1 librados: setting wanted keys 2024-03-25T18:13:44.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.263+0000 7f464f072640 1 librados: calling monclient init 2024-03-25T18:13:44.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.263+0000 7f464f072640 10 monclient: init 2024-03-25T18:13:44.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.264+0000 7f464f072640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:44.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.264+0000 7f464f072640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:44.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.264+0000 7f464f072640 10 monclient(hunting): picked mon.noname-a con 0x7f46480b1730 addr v1:172.21.15.156:6789/0 2024-03-25T18:13:44.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.264+0000 7f464f072640 10 monclient(hunting): picked mon.noname-b con 0x7f46480b4d60 addr v1:172.21.15.194:6789/0 2024-03-25T18:13:44.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.264+0000 7f464f072640 10 monclient(hunting): picked mon.noname-c con 0x7f464814be90 addr v1:172.21.15.156:6790/0 2024-03-25T18:13:44.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.264+0000 7f464f072640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46481db3a0 con 0x7f46480b1730 2024-03-25T18:13:44.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.264+0000 7f464f072640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46481dae80 con 0x7f46480b4d60 2024-03-25T18:13:44.268 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.264+0000 7f464f072640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46481db010 con 0x7f464814be90 2024-03-25T18:13:44.268 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.264+0000 7f464f072640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:44.268 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.264+0000 7f464f072640 10 monclient(hunting): authenticate will time out at 7305.575175s 2024-03-25T18:13:44.298 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.297+0000 7f464ce0e640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f46480b1730 0x7f46481371a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:42420/0 (socket says 172.21.15.156:42420) 2024-03-25T18:13:44.299 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.297+0000 7f464ce0e640 1 -- 172.21.15.156:0/4239649426 learned_addr learned my addr 172.21.15.156:0/4239649426 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:13:44.310 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.310+0000 7f46457fa640 10 client.?.objecter ms_handle_connect 0x7f46480b4d60 2024-03-25T18:13:44.310 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.310+0000 7f46457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:44.312 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.312+0000 7f46457fa640 10 client.?.objecter ms_handle_connect 0x7f46480b1730 2024-03-25T18:13:44.313 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.312+0000 7f46457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:44.321 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.321+0000 7f46457fa640 10 client.?.objecter ms_handle_connect 0x7f464814be90 2024-03-25T18:13:44.321 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.321+0000 7f46457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:44.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.326+0000 7f46457fa640 1 -- 172.21.15.156:0/4239649426 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365284872 0 0) 0x7f46481dae80 con 0x7f46480b4d60 2024-03-25T18:13:44.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.326+0000 7f46457fa640 10 monclient(hunting): _negotiate 2024-03-25T18:13:44.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.326+0000 7f46457fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:44.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.326+0000 7f46457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:44.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.326+0000 7f46457fa640 10 monclient(hunting): my global_id is 4471 2024-03-25T18:13:44.328 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.326+0000 7f46457fa640 1 -- 172.21.15.156:0/4239649426 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f462c004eb0 con 0x7f46480b4d60 2024-03-25T18:13:44.349 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.348+0000 7f46457fa640 1 -- 172.21.15.156:0/4239649426 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3460295555 0 0) 0x7f462c004eb0 con 0x7f46480b4d60 2024-03-25T18:13:44.349 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.348+0000 7f46457fa640 1 -- 172.21.15.156:0/4239649426 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46481dae80 con 0x7f46480b4d60 2024-03-25T18:13:44.349 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.348+0000 7f46457fa640 1 -- 172.21.15.156:0/4239649426 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f4634001870 con 0x7f46480b4d60 2024-03-25T18:13:44.349 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.348+0000 7f46457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:44.349 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.348+0000 7f46457fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:44.349 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.348+0000 7f46457fa640 10 monclient(hunting): dump: 2024-03-25T18:13:44.350 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:44.350 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:44.350 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:44.350 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:44.350 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:44.350 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:44.350 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:44.351 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:44.351 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:44.351 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:44.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.348+0000 7f46457fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:44.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.349+0000 7f46457fa640 1 -- 172.21.15.156:0/4239649426 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3259695840 0 0) 0x7f46481db3a0 con 0x7f46480b1730 2024-03-25T18:13:44.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.349+0000 7f46457fa640 10 monclient(hunting): _negotiate 2024-03-25T18:13:44.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.349+0000 7f46457fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:44.352 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.349+0000 7f46457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:44.352 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.349+0000 7f46457fa640 10 monclient(hunting): my global_id is 4467 2024-03-25T18:13:44.352 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.349+0000 7f46457fa640 1 -- 172.21.15.156:0/4239649426 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f462c004eb0 con 0x7f46480b1730 2024-03-25T18:13:44.356 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.355+0000 7f46457fa640 1 -- 172.21.15.156:0/4239649426 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488314111 0 0) 0x7f46481db010 con 0x7f464814be90 2024-03-25T18:13:44.356 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.355+0000 7f46457fa640 10 monclient(hunting): _negotiate 2024-03-25T18:13:44.356 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.355+0000 7f46457fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:44.356 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.355+0000 7f46457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:44.356 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.355+0000 7f46457fa640 10 monclient(hunting): my global_id is 4475 2024-03-25T18:13:44.356 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.355+0000 7f46457fa640 1 -- 172.21.15.156:0/4239649426 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46481db3a0 con 0x7f464814be90 2024-03-25T18:13:44.375 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.374+0000 7f46457fa640 1 -- 172.21.15.156:0/4239649426 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 958697798 0 0) 0x7f46481db3a0 con 0x7f464814be90 2024-03-25T18:13:44.375 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.374+0000 7f46457fa640 1 -- 172.21.15.156:0/4239649426 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46481db010 con 0x7f464814be90 2024-03-25T18:13:44.375 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.374+0000 7f46457fa640 1 -- 172.21.15.156:0/4239649426 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f4640001d50 con 0x7f464814be90 2024-03-25T18:13:44.375 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.374+0000 7f46457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:44.375 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.374+0000 7f46457fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:44.375 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.374+0000 7f46457fa640 10 monclient(hunting): dump: 2024-03-25T18:13:44.375 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:44.375 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:44.375 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:44.375 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:44.376 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:44.376 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:44.376 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:44.376 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:44.376 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:44.376 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:44.377 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.375+0000 7f46457fa640 1 -- 172.21.15.156:0/4239649426 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1673641755 0 0) 0x7f46481dae80 con 0x7f46480b4d60 2024-03-25T18:13:44.377 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.375+0000 7f46457fa640 1 -- 172.21.15.156:0/4239649426 >> v1:172.21.15.156:6790/0 conn(0x7f464814be90 legacy=0x7f46481d7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:44.377 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.375+0000 7f46457fa640 1 -- 172.21.15.156:0/4239649426 >> v1:172.21.15.156:6789/0 conn(0x7f46480b1730 legacy=0x7f46481371a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:44.377 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.375+0000 7f46457fa640 10 monclient: _finish_hunting 0 2024-03-25T18:13:44.377 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.375+0000 7f46457fa640 1 monclient: found mon.b 2024-03-25T18:13:44.377 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.375+0000 7f46457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:44.377 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.375+0000 7f46457fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:13:44.377 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.375+0000 7f46457fa640 1 -- 172.21.15.156:0/4239649426 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46481db8c0 con 0x7f46480b4d60 2024-03-25T18:13:44.377 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.375+0000 7f46457fa640 10 monclient: _finish_auth 0 2024-03-25T18:13:44.377 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.375+0000 7f46457fa640 10 monclient: _check_auth_tickets 2024-03-25T18:13:44.377 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.375+0000 7f46457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:44.378 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.375+0000 7f464f072640 5 monclient: authenticate success, global_id 4471 2024-03-25T18:13:44.378 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.375+0000 7f464f072640 10 monclient: _renew_subs 2024-03-25T18:13:44.378 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.375+0000 7f464f072640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:13:44.378 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.375+0000 7f464f072640 1 -- 172.21.15.156:0/4239649426 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46481dbb70 con 0x7f46480b4d60 2024-03-25T18:13:44.378 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.376+0000 7f464f072640 10 client.4471.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:44.378 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.376+0000 7f464f072640 10 monclient: _renew_subs 2024-03-25T18:13:44.378 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.376+0000 7f464f072640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:13:44.378 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.376+0000 7f464f072640 1 -- 172.21.15.156:0/4239649426 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f464813e2e0 con 0x7f46480b4d60 2024-03-25T18:13:44.378 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.376+0000 7f464f072640 1 librados: init done 2024-03-25T18:13:44.379 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.377+0000 7f464f072640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:44.379 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.377+0000 7f464f072640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:44.379 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.377+0000 7f464f072640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:13:44.379 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.377+0000 7f464f072640 1 -- 172.21.15.156:0/4239649426 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46481db530 con 0x7f46480b4d60 2024-03-25T18:13:44.399 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.398+0000 7f46457fa640 1 -- 172.21.15.156:0/4239649426 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46340035f0 con 0x7f46480b4d60 2024-03-25T18:13:44.399 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.398+0000 7f46457fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:44.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.398+0000 7f46457fa640 1 -- 172.21.15.156:0/4239649426 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f4634003900 con 0x7f46480b4d60 2024-03-25T18:13:44.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.398+0000 7f46457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:44.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.398+0000 7f46457fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:44.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.398+0000 7f46457fa640 10 monclient: dump: 2024-03-25T18:13:44.400 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:44.400 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:44.400 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:44.400 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:44.400 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:44.400 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:44.400 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:44.400 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:44.400 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:44.400 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:44.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.398+0000 7f4646ffd640 4 set_mon_vals no callback set 2024-03-25T18:13:44.401 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.399+0000 7f46457fa640 1 -- 172.21.15.156:0/4239649426 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f46340198d0 con 0x7f46480b4d60 2024-03-25T18:13:44.401 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.400+0000 7f464ce0e640 10 monclient: get_auth_request con 0x7f462c06df80 auth_method 0 2024-03-25T18:13:44.401 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.400+0000 7f46457fa640 10 client.4471.objecter ms_handle_connect 0x7f462c06df80 2024-03-25T18:13:44.404 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.403+0000 7f46457fa640 1 -- 172.21.15.156:0/4239649426 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f46340535a0 con 0x7f46480b4d60 2024-03-25T18:13:44.404 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.403+0000 7f46457fa640 10 client.4471.objecter ms_dispatch 0x7f4648000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:44.405 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.403+0000 7f46457fa640 3 client.4471.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:44.405 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.403+0000 7f46457fa640 3 client.4471.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:44.405 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.403+0000 7f46457fa640 20 client.4471.objecter dump_active .. 0 homeless 2024-03-25T18:13:44.683 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.682+0000 7f46457fa640 1 -- 172.21.15.156:0/4239649426 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f463404eee0 con 0x7f46480b4d60 2024-03-25T18:13:44.683 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.682+0000 7f46457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:44.683 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.682+0000 7f46457fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:44.938 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.937+0000 7f464f072640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:44.938 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.937+0000 7f464f072640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:44.939 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.937+0000 7f464f072640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:13:44.939 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.937+0000 7f464f072640 1 -- 172.21.15.156:0/4239649426 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46481db1a0 con 0x7f46480b4d60 2024-03-25T18:13:44.974 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.973+0000 7f46457fa640 1 -- 172.21.15.156:0/4239649426 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f4634050fe0 con 0x7f46480b4d60 2024-03-25T18:13:44.974 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.973+0000 7f46457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:44.974 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.973+0000 7f46457fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:44.974 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:13:44.975 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:44.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.976+0000 7f464f072640 10 librados: watch_flush enter 2024-03-25T18:13:44.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.976+0000 7f464f072640 10 librados: watch_flush exit 2024-03-25T18:13:44.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.976+0000 7f464f072640 20 client.4471.objecter shutdown clearing up homeless session... 2024-03-25T18:13:44.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.976+0000 7f464f072640 10 client.4471.objecter successfully canceled tick 2024-03-25T18:13:44.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.976+0000 7f464f072640 1 -- 172.21.15.156:0/4239649426 >> v1:172.21.15.194:6800/2946427713 conn(0x7f462c06df80 legacy=0x7f462c070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:44.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.976+0000 7f464f072640 10 monclient: shutdown 2024-03-25T18:13:44.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.976+0000 7f464f072640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:44.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.976+0000 7f464f072640 1 -- 172.21.15.156:0/4239649426 >> v1:172.21.15.194:6789/0 conn(0x7f46480b4d60 legacy=0x7f46481362e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:44.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.976+0000 7f464f072640 1 -- 172.21.15.156:0/4239649426 shutdown_connections 2024-03-25T18:13:44.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.976+0000 7f464f072640 1 -- 172.21.15.156:0/4239649426 >> 172.21.15.156:0/4239649426 conn(0x7f46480617a0 msgr2=0x7f464813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:44.978 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.976+0000 7f464f072640 1 -- 172.21.15.156:0/4239649426 shutdown_connections 2024-03-25T18:13:44.979 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.976+0000 7f464f072640 1 -- 172.21.15.156:0/4239649426 wait complete. 2024-03-25T18:13:44.979 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:44.976+0000 7f464f072640 1 librados: shutdown 2024-03-25T18:13:44.989 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:51.796 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:51.913 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.912+0000 7f7d26e0f640 10 monclient: get_monmap_and_config 2024-03-25T18:13:51.914 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.912+0000 7f7d26e0f640 10 monclient: build_initial_monmap 2024-03-25T18:13:51.914 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.912+0000 7f7d26e0f640 10 monclient: monmap: 2024-03-25T18:13:51.914 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:13:51.914 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:51.914 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:13:51.913901+0000 2024-03-25T18:13:51.914 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:13:51.913901+0000 2024-03-25T18:13:51.914 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:51.914 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:51.914 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:13:51.914 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:13:51.914 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:13:51.914 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:51.916 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.913+0000 7f7d26e0f640 1 Processor -- start 2024-03-25T18:13:51.916 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.913+0000 7f7d26e0f640 1 -- start start 2024-03-25T18:13:51.916 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.913+0000 7f7d26e0f640 10 monclient: init 2024-03-25T18:13:51.916 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.914+0000 7f7d26e0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:51.916 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.914+0000 7f7d26e0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:13:51.916 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.914+0000 7f7d26e0f640 10 monclient(hunting): picked mon.noname-c con 0x7f7d200a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:13:51.916 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.914+0000 7f7d26e0f640 10 monclient(hunting): picked mon.noname-b con 0x7f7d2013f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:13:51.916 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.914+0000 7f7d26e0f640 10 monclient(hunting): picked mon.noname-a con 0x7f7d201430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:13:51.916 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.914+0000 7f7d26e0f640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d20142520 con 0x7f7d201430c0 2024-03-25T18:13:51.916 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.914+0000 7f7d26e0f640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d200b28a0 con 0x7f7d200a98f0 2024-03-25T18:13:51.916 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.914+0000 7f7d26e0f640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d200b2a30 con 0x7f7d2013f710 2024-03-25T18:13:51.916 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.914+0000 7f7d26e0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:51.916 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.914+0000 7f7d26e0f640 10 monclient(hunting): authenticate will time out at 7313.225402s 2024-03-25T18:13:51.926 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.926+0000 7f7d24bab640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f7d200a98f0 0x7f7d200a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:48674/0 (socket says 172.21.15.156:48674) 2024-03-25T18:13:51.927 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.926+0000 7f7d24bab640 1 -- 172.21.15.156:0/789598385 learned_addr learned my addr 172.21.15.156:0/789598385 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:13:51.939 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.938+0000 7f7d1f7fe640 1 -- 172.21.15.156:0/789598385 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1412636442 0 0) 0x7f7d200b28a0 con 0x7f7d200a98f0 2024-03-25T18:13:51.939 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.938+0000 7f7d1f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:51.939 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.938+0000 7f7d1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:51.939 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.938+0000 7f7d1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:51.939 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.938+0000 7f7d1f7fe640 10 monclient(hunting): my global_id is 4474 2024-03-25T18:13:51.939 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.938+0000 7f7d1f7fe640 1 -- 172.21.15.156:0/789598385 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d10003520 con 0x7f7d200a98f0 2024-03-25T18:13:51.947 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.946+0000 7f7d1f7fe640 1 -- 172.21.15.156:0/789598385 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576773295 0 0) 0x7f7d20142520 con 0x7f7d201430c0 2024-03-25T18:13:51.947 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.946+0000 7f7d1f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:51.947 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.946+0000 7f7d1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:51.948 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.946+0000 7f7d1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:51.948 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.946+0000 7f7d1f7fe640 10 monclient(hunting): my global_id is 4470 2024-03-25T18:13:51.948 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.946+0000 7f7d1f7fe640 1 -- 172.21.15.156:0/789598385 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d200b28a0 con 0x7f7d201430c0 2024-03-25T18:13:51.948 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.947+0000 7f7d1f7fe640 1 -- 172.21.15.156:0/789598385 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4287853515 0 0) 0x7f7d200b2a30 con 0x7f7d2013f710 2024-03-25T18:13:51.948 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.947+0000 7f7d1f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:51.949 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.947+0000 7f7d1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:51.949 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.947+0000 7f7d1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:51.949 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.947+0000 7f7d1f7fe640 10 monclient(hunting): my global_id is 4478 2024-03-25T18:13:51.949 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.947+0000 7f7d1f7fe640 1 -- 172.21.15.156:0/789598385 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d20142520 con 0x7f7d2013f710 2024-03-25T18:13:51.960 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.958+0000 7f7d1f7fe640 1 -- 172.21.15.156:0/789598385 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 637921444 0 0) 0x7f7d10003520 con 0x7f7d200a98f0 2024-03-25T18:13:51.960 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.959+0000 7f7d1f7fe640 1 -- 172.21.15.156:0/789598385 >> v1:172.21.15.156:6790/0 conn(0x7f7d2013f710 legacy=0x7f7d201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:51.960 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.959+0000 7f7d1f7fe640 1 -- 172.21.15.156:0/789598385 >> v1:172.21.15.156:6789/0 conn(0x7f7d201430c0 legacy=0x7f7d2014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:51.960 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.959+0000 7f7d1f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:13:51.960 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.959+0000 7f7d1f7fe640 1 monclient: found mon.noname-c 2024-03-25T18:13:51.960 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.959+0000 7f7d1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:51.960 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.959+0000 7f7d1f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:13:51.960 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.959+0000 7f7d1f7fe640 1 -- 172.21.15.156:0/789598385 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d200b44c0 con 0x7f7d200a98f0 2024-03-25T18:13:51.960 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.959+0000 7f7d1f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:13:51.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.959+0000 7f7d1f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:13:51.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.959+0000 7f7d1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:51.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.959+0000 7f7d1f7fe640 1 -- 172.21.15.156:0/789598385 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7d14001690 con 0x7f7d200a98f0 2024-03-25T18:13:51.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.959+0000 7f7d26e0f640 5 monclient: authenticate success, global_id 4474 2024-03-25T18:13:51.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.959+0000 7f7d26e0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:51.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.959+0000 7f7d1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:51.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.959+0000 7f7d1f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:51.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.959+0000 7f7d1f7fe640 10 monclient: dump: 2024-03-25T18:13:51.961 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:51.961 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:51.961 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:51.961 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:51.961 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:51.961 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:51.962 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:51.962 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:51.962 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:51.962 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:51.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.959+0000 7f7d26e0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:51.984 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.983+0000 7f7d1f7fe640 1 -- 172.21.15.156:0/789598385 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d140017f0 con 0x7f7d200a98f0 2024-03-25T18:13:51.984 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.983+0000 7f7d1f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:51.984 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.983+0000 7f7d1f7fe640 1 -- 172.21.15.156:0/789598385 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7d14001c00 con 0x7f7d200a98f0 2024-03-25T18:13:51.984 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.983+0000 7f7d26e0f640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:51.984 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.983+0000 7f7d26e0f640 4 set_mon_vals no callback set 2024-03-25T18:13:51.984 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.983+0000 7f7d1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:51.984 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.983+0000 7f7d1f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:51.984 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.983+0000 7f7d1f7fe640 10 monclient: dump: 2024-03-25T18:13:51.985 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:51.985 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:51.985 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:51.985 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:51.985 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:51.985 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:51.985 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:51.985 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:51.985 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:51.985 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:51.986 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.983+0000 7f7d26e0f640 10 monclient: shutdown 2024-03-25T18:13:51.986 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.983+0000 7f7d26e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:51.987 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.983+0000 7f7d26e0f640 1 -- 172.21.15.156:0/789598385 >> v1:172.21.15.194:6789/0 conn(0x7f7d200a98f0 legacy=0x7f7d200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:51.987 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.984+0000 7f7d26e0f640 1 -- 172.21.15.156:0/789598385 shutdown_connections 2024-03-25T18:13:51.987 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.984+0000 7f7d26e0f640 1 -- 172.21.15.156:0/789598385 >> 172.21.15.156:0/789598385 conn(0x7f7d200b7ea0 msgr2=0x7f7d200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:51.987 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.984+0000 7f7d26e0f640 1 -- 172.21.15.156:0/789598385 shutdown_connections 2024-03-25T18:13:51.987 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.984+0000 7f7d26e0f640 1 -- 172.21.15.156:0/789598385 wait complete. 2024-03-25T18:13:51.987 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.984+0000 7f7d26e0f640 10 monclient: build_initial_monmap 2024-03-25T18:13:51.987 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.984+0000 7f7d26e0f640 10 monclient: monmap: 2024-03-25T18:13:51.987 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:13:51.987 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:51.987 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:13:51.987 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:13:51.987 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:51.987 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:13:51.987 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:13:51.988 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:13:51.988 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:13:51.988 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:51.988 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.985+0000 7f7d26e0f640 1 librados: starting msgr at 2024-03-25T18:13:51.988 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.985+0000 7f7d26e0f640 1 librados: starting objecter 2024-03-25T18:13:51.988 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.985+0000 7f7d26e0f640 1 Processor -- start 2024-03-25T18:13:51.988 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.985+0000 7f7d26e0f640 1 -- start start 2024-03-25T18:13:51.988 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.985+0000 7f7d26e0f640 1 librados: setting wanted keys 2024-03-25T18:13:51.988 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.985+0000 7f7d26e0f640 1 librados: calling monclient init 2024-03-25T18:13:51.988 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.985+0000 7f7d26e0f640 10 monclient: init 2024-03-25T18:13:51.989 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.986+0000 7f7d26e0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:51.989 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.986+0000 7f7d26e0f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:51.989 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.986+0000 7f7d26e0f640 10 monclient(hunting): picked mon.noname-c con 0x7f7d200a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:13:51.989 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.986+0000 7f7d26e0f640 10 monclient(hunting): picked mon.noname-b con 0x7f7d2013f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:13:51.989 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.986+0000 7f7d26e0f640 10 monclient(hunting): picked mon.noname-a con 0x7f7d201430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:13:51.990 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.986+0000 7f7d26e0f640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d20139a40 con 0x7f7d201430c0 2024-03-25T18:13:51.990 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.986+0000 7f7d26e0f640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d20133f20 con 0x7f7d2013f710 2024-03-25T18:13:51.990 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.986+0000 7f7d26e0f640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d200ab950 con 0x7f7d200a98f0 2024-03-25T18:13:51.990 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.986+0000 7f7d26e0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:51.990 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.986+0000 7f7d26e0f640 10 monclient(hunting): authenticate will time out at 7313.297331s 2024-03-25T18:13:51.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.994+0000 7f7d253ac640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f7d201430c0 0x7f7d20133810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:48782/0 (socket says 172.21.15.156:48782) 2024-03-25T18:13:51.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.994+0000 7f7d253ac640 1 -- 172.21.15.156:0/1794838767 learned_addr learned my addr 172.21.15.156:0/1794838767 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:13:51.999 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.999+0000 7f7d1d7fa640 10 client.?.objecter ms_handle_connect 0x7f7d201430c0 2024-03-25T18:13:51.999 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:51.999+0000 7f7d1d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:52.003 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.003+0000 7f7d1d7fa640 10 client.?.objecter ms_handle_connect 0x7f7d2013f710 2024-03-25T18:13:52.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.003+0000 7f7d1d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:52.005 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.004+0000 7f7d1d7fa640 1 -- 172.21.15.156:0/1794838767 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302850772 0 0) 0x7f7d20139a40 con 0x7f7d201430c0 2024-03-25T18:13:52.005 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.004+0000 7f7d1d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:13:52.005 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.004+0000 7f7d1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:52.005 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.004+0000 7f7d1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:52.005 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.004+0000 7f7d1d7fa640 10 monclient(hunting): my global_id is 4473 2024-03-25T18:13:52.005 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.004+0000 7f7d1d7fa640 1 -- 172.21.15.156:0/1794838767 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d04004eb0 con 0x7f7d201430c0 2024-03-25T18:13:52.011 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.010+0000 7f7d1d7fa640 10 client.?.objecter ms_handle_connect 0x7f7d200a98f0 2024-03-25T18:13:52.011 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.010+0000 7f7d1d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:52.013 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.012+0000 7f7d1d7fa640 1 -- 172.21.15.156:0/1794838767 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1992410028 0 0) 0x7f7d20133f20 con 0x7f7d2013f710 2024-03-25T18:13:52.013 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.012+0000 7f7d1d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:13:52.013 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.012+0000 7f7d1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:52.013 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.012+0000 7f7d1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:52.013 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.012+0000 7f7d1d7fa640 10 monclient(hunting): my global_id is 4477 2024-03-25T18:13:52.013 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.012+0000 7f7d1d7fa640 1 -- 172.21.15.156:0/1794838767 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d20139a40 con 0x7f7d2013f710 2024-03-25T18:13:52.016 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.015+0000 7f7d1d7fa640 1 -- 172.21.15.156:0/1794838767 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3904730243 0 0) 0x7f7d04004eb0 con 0x7f7d201430c0 2024-03-25T18:13:52.016 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.015+0000 7f7d1d7fa640 1 -- 172.21.15.156:0/1794838767 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d20133f20 con 0x7f7d201430c0 2024-03-25T18:13:52.016 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.015+0000 7f7d1d7fa640 1 -- 172.21.15.156:0/1794838767 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7d18001c90 con 0x7f7d201430c0 2024-03-25T18:13:52.016 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.015+0000 7f7d1d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:52.016 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.015+0000 7f7d1d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:52.016 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.015+0000 7f7d1d7fa640 10 monclient(hunting): dump: 2024-03-25T18:13:52.016 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:52.016 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:52.017 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:52.017 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:52.017 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:52.017 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:52.017 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:52.017 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:52.017 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:52.017 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:52.017 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.015+0000 7f7d1d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:52.022 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.021+0000 7f7d1d7fa640 1 -- 172.21.15.156:0/1794838767 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3692874503 0 0) 0x7f7d200ab950 con 0x7f7d200a98f0 2024-03-25T18:13:52.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.022+0000 7f7d1d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:13:52.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.022+0000 7f7d1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:52.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.022+0000 7f7d1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:52.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.022+0000 7f7d1d7fa640 10 monclient(hunting): my global_id is 4481 2024-03-25T18:13:52.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.022+0000 7f7d1d7fa640 1 -- 172.21.15.156:0/1794838767 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d04004eb0 con 0x7f7d200a98f0 2024-03-25T18:13:52.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.024+0000 7f7d1d7fa640 1 -- 172.21.15.156:0/1794838767 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1267051427 0 0) 0x7f7d20133f20 con 0x7f7d201430c0 2024-03-25T18:13:52.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.024+0000 7f7d1d7fa640 1 -- 172.21.15.156:0/1794838767 >> v1:172.21.15.156:6790/0 conn(0x7f7d200a98f0 legacy=0x7f7d20138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:52.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.024+0000 7f7d1d7fa640 1 -- 172.21.15.156:0/1794838767 >> v1:172.21.15.194:6789/0 conn(0x7f7d2013f710 legacy=0x7f7d20133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:52.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.024+0000 7f7d1d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:13:52.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.024+0000 7f7d1d7fa640 1 monclient: found mon.a 2024-03-25T18:13:52.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.024+0000 7f7d1d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:52.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.024+0000 7f7d1d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:13:52.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.024+0000 7f7d1d7fa640 1 -- 172.21.15.156:0/1794838767 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d200abc30 con 0x7f7d201430c0 2024-03-25T18:13:52.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.024+0000 7f7d1d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:13:52.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.024+0000 7f7d1d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:13:52.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.024+0000 7f7d1d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:52.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.024+0000 7f7d26e0f640 5 monclient: authenticate success, global_id 4473 2024-03-25T18:13:52.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.024+0000 7f7d26e0f640 10 monclient: _renew_subs 2024-03-25T18:13:52.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.024+0000 7f7d26e0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:13:52.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.024+0000 7f7d26e0f640 1 -- 172.21.15.156:0/1794838767 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d200abf10 con 0x7f7d201430c0 2024-03-25T18:13:52.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.025+0000 7f7d26e0f640 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:52.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.025+0000 7f7d26e0f640 10 monclient: _renew_subs 2024-03-25T18:13:52.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.025+0000 7f7d26e0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:13:52.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.025+0000 7f7d26e0f640 1 -- 172.21.15.156:0/1794838767 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d20139370 con 0x7f7d201430c0 2024-03-25T18:13:52.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.025+0000 7f7d26e0f640 1 librados: init done 2024-03-25T18:13:52.028 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.025+0000 7f7d26e0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:52.028 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.025+0000 7f7d26e0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:52.028 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.025+0000 7f7d26e0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:13:52.028 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.025+0000 7f7d26e0f640 1 -- 172.21.15.156:0/1794838767 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d200627b0 con 0x7f7d201430c0 2024-03-25T18:13:52.319 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.318+0000 7f7d1d7fa640 1 -- 172.21.15.156:0/1794838767 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d18003970 con 0x7f7d201430c0 2024-03-25T18:13:52.319 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.318+0000 7f7d1d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:52.319 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.318+0000 7f7d1d7fa640 1 -- 172.21.15.156:0/1794838767 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7d18003d80 con 0x7f7d201430c0 2024-03-25T18:13:52.319 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.318+0000 7f7d1d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:52.320 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.318+0000 7f7d1d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:52.320 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.318+0000 7f7d1d7fa640 10 monclient: dump: 2024-03-25T18:13:52.320 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:52.320 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:52.320 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:52.320 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:52.320 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:52.320 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:52.320 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:52.320 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:52.320 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:52.321 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:52.321 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.318+0000 7f7d1effd640 4 set_mon_vals no callback set 2024-03-25T18:13:52.321 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.319+0000 7f7d1d7fa640 1 -- 172.21.15.156:0/1794838767 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f7d18003fa0 con 0x7f7d201430c0 2024-03-25T18:13:52.321 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.319+0000 7f7d24bab640 10 monclient: get_auth_request con 0x7f7d0406de00 auth_method 0 2024-03-25T18:13:52.321 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.320+0000 7f7d1d7fa640 10 client.4473.objecter ms_handle_connect 0x7f7d0406de00 2024-03-25T18:13:52.325 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.324+0000 7f7d1d7fa640 1 -- 172.21.15.156:0/1794838767 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f7d18054a60 con 0x7f7d201430c0 2024-03-25T18:13:52.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.324+0000 7f7d1d7fa640 10 client.4473.objecter ms_dispatch 0x7f7d20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:52.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.324+0000 7f7d1d7fa640 3 client.4473.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:52.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.324+0000 7f7d1d7fa640 3 client.4473.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:52.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.324+0000 7f7d1d7fa640 20 client.4473.objecter dump_active .. 0 homeless 2024-03-25T18:13:52.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.324+0000 7f7d1d7fa640 1 -- 172.21.15.156:0/1794838767 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7d18050570 con 0x7f7d201430c0 2024-03-25T18:13:52.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.324+0000 7f7d1d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:52.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.324+0000 7f7d1d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:52.557 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.556+0000 7f7d26e0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:52.557 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.556+0000 7f7d26e0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:52.557 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.556+0000 7f7d26e0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:13:52.557 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.556+0000 7f7d26e0f640 1 -- 172.21.15.156:0/1794838767 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d200b7180 con 0x7f7d201430c0 2024-03-25T18:13:52.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.588+0000 7f7d1d7fa640 1 -- 172.21.15.156:0/1794838767 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f7d180524a0 con 0x7f7d201430c0 2024-03-25T18:13:52.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.588+0000 7f7d1d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:52.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.588+0000 7f7d1d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:52.590 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:13:52.590 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:52.592 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.590+0000 7f7d26e0f640 10 librados: watch_flush enter 2024-03-25T18:13:52.592 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.590+0000 7f7d26e0f640 10 librados: watch_flush exit 2024-03-25T18:13:52.592 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.590+0000 7f7d26e0f640 20 client.4473.objecter shutdown clearing up homeless session... 2024-03-25T18:13:52.592 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.590+0000 7f7d26e0f640 10 client.4473.objecter successfully canceled tick 2024-03-25T18:13:52.592 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.590+0000 7f7d26e0f640 1 -- 172.21.15.156:0/1794838767 >> v1:172.21.15.194:6800/2946427713 conn(0x7f7d0406de00 legacy=0x7f7d040702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:52.592 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.590+0000 7f7d26e0f640 10 monclient: shutdown 2024-03-25T18:13:52.592 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.590+0000 7f7d26e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:52.592 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.590+0000 7f7d26e0f640 1 -- 172.21.15.156:0/1794838767 >> v1:172.21.15.156:6789/0 conn(0x7f7d201430c0 legacy=0x7f7d20133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:52.592 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.590+0000 7f7d26e0f640 1 -- 172.21.15.156:0/1794838767 shutdown_connections 2024-03-25T18:13:52.592 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.590+0000 7f7d26e0f640 1 -- 172.21.15.156:0/1794838767 >> 172.21.15.156:0/1794838767 conn(0x7f7d200b7ea0 msgr2=0x7f7d20074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:52.592 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.590+0000 7f7d26e0f640 1 -- 172.21.15.156:0/1794838767 shutdown_connections 2024-03-25T18:13:52.592 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.590+0000 7f7d26e0f640 1 -- 172.21.15.156:0/1794838767 wait complete. 2024-03-25T18:13:52.593 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:52.591+0000 7f7d26e0f640 1 librados: shutdown 2024-03-25T18:13:52.602 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:59.407 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:59.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.524+0000 7fcccb3e6640 10 monclient: get_monmap_and_config 2024-03-25T18:13:59.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.524+0000 7fcccb3e6640 10 monclient: build_initial_monmap 2024-03-25T18:13:59.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.525+0000 7fcccb3e6640 10 monclient: monmap: 2024-03-25T18:13:59.526 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:13:59.526 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:59.526 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:13:59.525977+0000 2024-03-25T18:13:59.526 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:13:59.525977+0000 2024-03-25T18:13:59.526 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:59.526 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:59.526 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:13:59.527 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:13:59.527 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:13:59.527 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:59.528 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.525+0000 7fcccb3e6640 1 Processor -- start 2024-03-25T18:13:59.528 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.525+0000 7fcccb3e6640 1 -- start start 2024-03-25T18:13:59.528 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.525+0000 7fcccb3e6640 10 monclient: init 2024-03-25T18:13:59.528 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.526+0000 7fcccb3e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:59.528 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.526+0000 7fcccb3e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:59.529 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.526+0000 7fcccb3e6640 10 monclient(hunting): picked mon.noname-c con 0x7fccc40a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:13:59.529 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.526+0000 7fcccb3e6640 10 monclient(hunting): picked mon.noname-a con 0x7fccc413f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:13:59.529 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.526+0000 7fcccb3e6640 10 monclient(hunting): picked mon.noname-b con 0x7fccc41430c0 addr v1:172.21.15.156:6790/0 2024-03-25T18:13:59.529 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.526+0000 7fcccb3e6640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc4142520 con 0x7fccc413f710 2024-03-25T18:13:59.529 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.526+0000 7fcccb3e6640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc40b28a0 con 0x7fccc40a98f0 2024-03-25T18:13:59.529 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.526+0000 7fcccb3e6640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc40b2a30 con 0x7fccc41430c0 2024-03-25T18:13:59.529 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.526+0000 7fcccb3e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:59.529 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.526+0000 7fcccb3e6640 10 monclient(hunting): authenticate will time out at 7320.837642s 2024-03-25T18:13:59.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.534+0000 7fccc9983640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7fccc41430c0 0x7fccc414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:47236/0 (socket says 172.21.15.156:47236) 2024-03-25T18:13:59.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.534+0000 7fccc9983640 1 -- 172.21.15.156:0/3418759994 learned_addr learned my addr 172.21.15.156:0/3418759994 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:13:59.546 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.545+0000 7fccbbfff640 1 -- 172.21.15.156:0/3418759994 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1337087928 0 0) 0x7fccc40b2a30 con 0x7fccc41430c0 2024-03-25T18:13:59.546 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.545+0000 7fccbbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:59.546 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.545+0000 7fccbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:59.546 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.545+0000 7fccbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:59.546 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.545+0000 7fccbbfff640 10 monclient(hunting): my global_id is 4484 2024-03-25T18:13:59.546 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.545+0000 7fccbbfff640 1 -- 172.21.15.156:0/3418759994 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccb0003520 con 0x7fccc41430c0 2024-03-25T18:13:59.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.546+0000 7fccbbfff640 1 -- 172.21.15.156:0/3418759994 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683540905 0 0) 0x7fccc4142520 con 0x7fccc413f710 2024-03-25T18:13:59.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.546+0000 7fccbbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:59.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.546+0000 7fccbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:59.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.546+0000 7fccbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:59.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.546+0000 7fccbbfff640 10 monclient(hunting): my global_id is 4476 2024-03-25T18:13:59.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.546+0000 7fccbbfff640 1 -- 172.21.15.156:0/3418759994 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc40b2a30 con 0x7fccc413f710 2024-03-25T18:13:59.557 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.556+0000 7fccbbfff640 1 -- 172.21.15.156:0/3418759994 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1103000557 0 0) 0x7fccc40b2a30 con 0x7fccc413f710 2024-03-25T18:13:59.557 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.556+0000 7fccbbfff640 1 -- 172.21.15.156:0/3418759994 >> v1:172.21.15.156:6790/0 conn(0x7fccc41430c0 legacy=0x7fccc414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:59.557 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.556+0000 7fccbbfff640 1 -- 172.21.15.156:0/3418759994 >> v1:172.21.15.194:6789/0 conn(0x7fccc40a98f0 legacy=0x7fccc40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:59.557 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.556+0000 7fccbbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:13:59.557 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.556+0000 7fccbbfff640 1 monclient: found mon.noname-a 2024-03-25T18:13:59.557 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.556+0000 7fccbbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:59.557 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.556+0000 7fccbbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:13:59.557 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.556+0000 7fccbbfff640 1 -- 172.21.15.156:0/3418759994 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc40b44c0 con 0x7fccc413f710 2024-03-25T18:13:59.557 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.556+0000 7fccbbfff640 10 monclient: _finish_auth 0 2024-03-25T18:13:59.558 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.556+0000 7fccbbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:13:59.558 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.556+0000 7fccbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:59.558 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.556+0000 7fccbbfff640 1 -- 172.21.15.156:0/3418759994 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fccb4001670 con 0x7fccc413f710 2024-03-25T18:13:59.558 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.556+0000 7fcccb3e6640 5 monclient: authenticate success, global_id 4476 2024-03-25T18:13:59.558 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.556+0000 7fcccb3e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:59.558 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.556+0000 7fccbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:59.558 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.556+0000 7fccbbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:59.558 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.556+0000 7fccbbfff640 10 monclient: dump: 2024-03-25T18:13:59.558 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:59.558 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:59.558 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:59.558 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:59.558 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:59.559 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:59.559 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:59.559 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:59.559 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:59.559 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:59.559 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.556+0000 7fcccb3e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:59.562 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.561+0000 7fccbbfff640 1 -- 172.21.15.156:0/3418759994 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fccb40017d0 con 0x7fccc413f710 2024-03-25T18:13:59.562 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.561+0000 7fccbbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:59.562 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.561+0000 7fccbbfff640 1 -- 172.21.15.156:0/3418759994 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fccb4001ba0 con 0x7fccc413f710 2024-03-25T18:13:59.562 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.561+0000 7fccbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:59.562 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.561+0000 7fccbbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:59.562 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.561+0000 7fccbbfff640 10 monclient: dump: 2024-03-25T18:13:59.562 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:59.562 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:59.562 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:59.562 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:59.562 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:59.562 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:59.563 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:59.563 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:59.563 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:59.563 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:59.563 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.561+0000 7fcccb3e6640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:59.563 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.561+0000 7fcccb3e6640 4 set_mon_vals no callback set 2024-03-25T18:13:59.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.561+0000 7fcccb3e6640 10 monclient: shutdown 2024-03-25T18:13:59.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.561+0000 7fcccb3e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:59.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.561+0000 7fcccb3e6640 1 -- 172.21.15.156:0/3418759994 >> v1:172.21.15.156:6789/0 conn(0x7fccc413f710 legacy=0x7fccc41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:59.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.561+0000 7fcccb3e6640 1 -- 172.21.15.156:0/3418759994 shutdown_connections 2024-03-25T18:13:59.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.561+0000 7fcccb3e6640 1 -- 172.21.15.156:0/3418759994 >> 172.21.15.156:0/3418759994 conn(0x7fccc40b7ea0 msgr2=0x7fccc40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:59.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.562+0000 7fcccb3e6640 1 -- 172.21.15.156:0/3418759994 shutdown_connections 2024-03-25T18:13:59.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.562+0000 7fcccb3e6640 1 -- 172.21.15.156:0/3418759994 wait complete. 2024-03-25T18:13:59.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.562+0000 7fcccb3e6640 10 monclient: build_initial_monmap 2024-03-25T18:13:59.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.562+0000 7fcccb3e6640 10 monclient: monmap: 2024-03-25T18:13:59.565 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:13:59.565 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:59.565 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:13:59.566 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:13:59.566 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:59.566 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:13:59.566 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:13:59.566 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:13:59.566 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:13:59.566 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:59.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.563+0000 7fcccb3e6640 1 librados: starting msgr at 2024-03-25T18:13:59.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.563+0000 7fcccb3e6640 1 librados: starting objecter 2024-03-25T18:13:59.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.563+0000 7fcccb3e6640 1 Processor -- start 2024-03-25T18:13:59.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.563+0000 7fcccb3e6640 1 -- start start 2024-03-25T18:13:59.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.563+0000 7fcccb3e6640 1 librados: setting wanted keys 2024-03-25T18:13:59.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.563+0000 7fcccb3e6640 1 librados: calling monclient init 2024-03-25T18:13:59.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.563+0000 7fcccb3e6640 10 monclient: init 2024-03-25T18:13:59.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.563+0000 7fcccb3e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:59.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.563+0000 7fcccb3e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:59.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.563+0000 7fcccb3e6640 10 monclient(hunting): picked mon.noname-b con 0x7fccc40a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:13:59.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.563+0000 7fcccb3e6640 10 monclient(hunting): picked mon.noname-a con 0x7fccc413f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:13:59.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.564+0000 7fcccb3e6640 10 monclient(hunting): picked mon.noname-c con 0x7fccc41430c0 addr v1:172.21.15.156:6790/0 2024-03-25T18:13:59.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.564+0000 7fcccb3e6640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc41da880 con 0x7fccc413f710 2024-03-25T18:13:59.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.564+0000 7fcccb3e6640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc41399a0 con 0x7fccc40a98f0 2024-03-25T18:13:59.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.564+0000 7fcccb3e6640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc41db700 con 0x7fccc41430c0 2024-03-25T18:13:59.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.564+0000 7fcccb3e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:59.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.564+0000 7fcccb3e6640 10 monclient(hunting): authenticate will time out at 7320.874839s 2024-03-25T18:13:59.572 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.571+0000 7fccc9182640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fccc40a98f0 0x7fccc41daff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:53490/0 (socket says 172.21.15.156:53490) 2024-03-25T18:13:59.572 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.571+0000 7fccc9182640 1 -- 172.21.15.156:0/1088868490 learned_addr learned my addr 172.21.15.156:0/1088868490 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:13:59.575 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.574+0000 7fccb9ffb640 10 client.?.objecter ms_handle_connect 0x7fccc40a98f0 2024-03-25T18:13:59.575 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.574+0000 7fccb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:59.583 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.582+0000 7fccb9ffb640 10 client.?.objecter ms_handle_connect 0x7fccc41430c0 2024-03-25T18:13:59.583 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.582+0000 7fccb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:59.584 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.583+0000 7fccb9ffb640 10 client.?.objecter ms_handle_connect 0x7fccc413f710 2024-03-25T18:13:59.584 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.583+0000 7fccb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:59.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.588+0000 7fccb9ffb640 1 -- 172.21.15.156:0/1088868490 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 616646573 0 0) 0x7fccc41399a0 con 0x7fccc40a98f0 2024-03-25T18:13:59.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.588+0000 7fccb9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:59.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.588+0000 7fccb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:59.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.588+0000 7fccb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:59.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.588+0000 7fccb9ffb640 10 monclient(hunting): my global_id is 4483 2024-03-25T18:13:59.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.588+0000 7fccb9ffb640 1 -- 172.21.15.156:0/1088868490 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcca8004eb0 con 0x7fccc40a98f0 2024-03-25T18:13:59.593 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.592+0000 7fccb9ffb640 1 -- 172.21.15.156:0/1088868490 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384293230 0 0) 0x7fccc41db700 con 0x7fccc41430c0 2024-03-25T18:13:59.593 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.592+0000 7fccb9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:59.593 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.592+0000 7fccb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:59.593 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.592+0000 7fccb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:59.593 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.592+0000 7fccb9ffb640 10 monclient(hunting): my global_id is 4487 2024-03-25T18:13:59.594 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.592+0000 7fccb9ffb640 1 -- 172.21.15.156:0/1088868490 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc41399a0 con 0x7fccc41430c0 2024-03-25T18:13:59.599 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.598+0000 7fccb9ffb640 1 -- 172.21.15.156:0/1088868490 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3990464721 0 0) 0x7fcca8004eb0 con 0x7fccc40a98f0 2024-03-25T18:13:59.599 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.598+0000 7fccb9ffb640 1 -- 172.21.15.156:0/1088868490 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccc41db700 con 0x7fccc40a98f0 2024-03-25T18:13:59.599 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.598+0000 7fccb9ffb640 1 -- 172.21.15.156:0/1088868490 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fccc0001a30 con 0x7fccc40a98f0 2024-03-25T18:13:59.600 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.598+0000 7fccb9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:59.600 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.598+0000 7fccb9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:59.600 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.598+0000 7fccb9ffb640 10 monclient(hunting): dump: 2024-03-25T18:13:59.600 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:59.600 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:59.600 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:59.600 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:59.600 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:59.600 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:59.600 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:59.600 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:59.600 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:59.600 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:59.600 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.598+0000 7fccb9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:59.604 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.603+0000 7fccb9ffb640 1 -- 172.21.15.156:0/1088868490 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 671093750 0 0) 0x7fccc41da880 con 0x7fccc413f710 2024-03-25T18:13:59.604 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.603+0000 7fccb9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:59.604 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.603+0000 7fccb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:59.604 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.603+0000 7fccb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:59.604 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.603+0000 7fccb9ffb640 10 monclient(hunting): my global_id is 4479 2024-03-25T18:13:59.604 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.603+0000 7fccb9ffb640 1 -- 172.21.15.156:0/1088868490 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcca8004eb0 con 0x7fccc413f710 2024-03-25T18:13:59.610 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.609+0000 7fccb9ffb640 1 -- 172.21.15.156:0/1088868490 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2519282627 0 0) 0x7fccc41399a0 con 0x7fccc41430c0 2024-03-25T18:13:59.611 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.610+0000 7fccb9ffb640 1 -- 172.21.15.156:0/1088868490 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccc41da880 con 0x7fccc41430c0 2024-03-25T18:13:59.611 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.610+0000 7fccb9ffb640 1 -- 172.21.15.156:0/1088868490 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fccbc001a70 con 0x7fccc41430c0 2024-03-25T18:13:59.611 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.610+0000 7fccb9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:59.611 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.610+0000 7fccb9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:59.611 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.610+0000 7fccb9ffb640 10 monclient(hunting): dump: 2024-03-25T18:13:59.611 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:59.611 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:59.611 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:59.611 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:59.611 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:59.611 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:59.611 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:59.611 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:59.611 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:59.612 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:59.615 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.613+0000 7fccb9ffb640 1 -- 172.21.15.156:0/1088868490 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2207367799 0 0) 0x7fccc41db700 con 0x7fccc40a98f0 2024-03-25T18:13:59.615 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.614+0000 7fccb9ffb640 1 -- 172.21.15.156:0/1088868490 >> v1:172.21.15.156:6790/0 conn(0x7fccc41430c0 legacy=0x7fccc41e1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:59.615 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.614+0000 7fccb9ffb640 1 -- 172.21.15.156:0/1088868490 >> v1:172.21.15.156:6789/0 conn(0x7fccc413f710 legacy=0x7fccc4138410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:59.615 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.614+0000 7fccb9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:13:59.615 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.614+0000 7fccb9ffb640 1 monclient: found mon.b 2024-03-25T18:13:59.615 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.614+0000 7fccb9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:59.615 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.614+0000 7fccb9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:13:59.616 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.614+0000 7fccb9ffb640 1 -- 172.21.15.156:0/1088868490 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc41daad0 con 0x7fccc40a98f0 2024-03-25T18:13:59.616 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.614+0000 7fccb9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:13:59.616 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.614+0000 7fccb9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:13:59.616 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.614+0000 7fccb9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:59.616 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.614+0000 7fcccb3e6640 5 monclient: authenticate success, global_id 4483 2024-03-25T18:13:59.616 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.614+0000 7fcccb3e6640 10 monclient: _renew_subs 2024-03-25T18:13:59.616 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.614+0000 7fcccb3e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:13:59.616 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.614+0000 7fcccb3e6640 1 -- 172.21.15.156:0/1088868490 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc4138b20 con 0x7fccc40a98f0 2024-03-25T18:13:59.616 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.614+0000 7fcccb3e6640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:59.616 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.614+0000 7fcccb3e6640 10 monclient: _renew_subs 2024-03-25T18:13:59.616 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.614+0000 7fcccb3e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:13:59.616 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.614+0000 7fcccb3e6640 1 -- 172.21.15.156:0/1088868490 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc4137a70 con 0x7fccc40a98f0 2024-03-25T18:13:59.616 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.614+0000 7fcccb3e6640 1 librados: init done 2024-03-25T18:13:59.617 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.615+0000 7fcccb3e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:59.617 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.615+0000 7fcccb3e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:59.617 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.615+0000 7fcccb3e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:13:59.617 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.615+0000 7fcccb3e6640 1 -- 172.21.15.156:0/1088868490 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc40627b0 con 0x7fccc40a98f0 2024-03-25T18:13:59.899 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.897+0000 7fccb9ffb640 1 -- 172.21.15.156:0/1088868490 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fccc0001c70 con 0x7fccc40a98f0 2024-03-25T18:13:59.899 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.897+0000 7fccb9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:59.899 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.897+0000 7fccb9ffb640 1 -- 172.21.15.156:0/1088868490 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fccc0003a60 con 0x7fccc40a98f0 2024-03-25T18:13:59.899 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.897+0000 7fccb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:59.899 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.897+0000 7fccb9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:59.899 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.897+0000 7fccb9ffb640 10 monclient: dump: 2024-03-25T18:13:59.899 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:13:59.899 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:13:59.899 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:59.899 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:13:59.899 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:59.899 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:13:59.899 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:13:59.900 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:13:59.900 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:13:59.900 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:13:59.900 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.897+0000 7fccbb7fe640 4 set_mon_vals no callback set 2024-03-25T18:13:59.900 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.898+0000 7fccb9ffb640 1 -- 172.21.15.156:0/1088868490 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7fccc0003c80 con 0x7fccc40a98f0 2024-03-25T18:13:59.900 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.899+0000 7fccc8981640 10 monclient: get_auth_request con 0x7fcca806dfb0 auth_method 0 2024-03-25T18:13:59.900 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.899+0000 7fccb9ffb640 10 client.4483.objecter ms_handle_connect 0x7fcca806dfb0 2024-03-25T18:13:59.905 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.904+0000 7fccb9ffb640 1 -- 172.21.15.156:0/1088868490 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7fccc0054830 con 0x7fccc40a98f0 2024-03-25T18:13:59.905 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.904+0000 7fccb9ffb640 10 client.4483.objecter ms_dispatch 0x7fccc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:59.905 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.904+0000 7fccb9ffb640 3 client.4483.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:59.905 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.904+0000 7fccb9ffb640 3 client.4483.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:59.905 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.904+0000 7fccb9ffb640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-25T18:13:59.905 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.904+0000 7fccb9ffb640 1 -- 172.21.15.156:0/1088868490 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fccc0050340 con 0x7fccc40a98f0 2024-03-25T18:13:59.905 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.904+0000 7fccb9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:59.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:13:59.904+0000 7fccb9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:00.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:00.152+0000 7fcccb3e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:00.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:00.152+0000 7fcccb3e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:00.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:00.152+0000 7fcccb3e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:14:00.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:00.152+0000 7fcccb3e6640 1 -- 172.21.15.156:0/1088868490 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fccc41dac60 con 0x7fccc40a98f0 2024-03-25T18:14:00.185 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:00.184+0000 7fccb9ffb640 1 -- 172.21.15.156:0/1088868490 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7fccc0052270 con 0x7fccc40a98f0 2024-03-25T18:14:00.185 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:00.184+0000 7fccb9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:00.185 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:00.184+0000 7fccb9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:00.185 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:14:00.186 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:00.187 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:00.186+0000 7fcccb3e6640 10 librados: watch_flush enter 2024-03-25T18:14:00.188 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:00.186+0000 7fcccb3e6640 10 librados: watch_flush exit 2024-03-25T18:14:00.188 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:00.186+0000 7fcccb3e6640 20 client.4483.objecter shutdown clearing up homeless session... 2024-03-25T18:14:00.188 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:00.186+0000 7fcccb3e6640 10 client.4483.objecter successfully canceled tick 2024-03-25T18:14:00.188 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:00.186+0000 7fcccb3e6640 1 -- 172.21.15.156:0/1088868490 >> v1:172.21.15.194:6800/2946427713 conn(0x7fcca806dfb0 legacy=0x7fcca8070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:00.188 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:00.186+0000 7fcccb3e6640 10 monclient: shutdown 2024-03-25T18:14:00.188 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:00.186+0000 7fcccb3e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:00.188 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:00.186+0000 7fcccb3e6640 1 -- 172.21.15.156:0/1088868490 >> v1:172.21.15.194:6789/0 conn(0x7fccc40a98f0 legacy=0x7fccc41daff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:00.188 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:00.186+0000 7fcccb3e6640 1 -- 172.21.15.156:0/1088868490 shutdown_connections 2024-03-25T18:14:00.188 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:00.186+0000 7fcccb3e6640 1 -- 172.21.15.156:0/1088868490 >> 172.21.15.156:0/1088868490 conn(0x7fccc40b7ea0 msgr2=0x7fccc413d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:00.188 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:00.186+0000 7fcccb3e6640 1 -- 172.21.15.156:0/1088868490 shutdown_connections 2024-03-25T18:14:00.188 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:00.187+0000 7fcccb3e6640 1 -- 172.21.15.156:0/1088868490 wait complete. 2024-03-25T18:14:00.188 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:00.187+0000 7fcccb3e6640 1 librados: shutdown 2024-03-25T18:14:00.200 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:04.142 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:06:10.132 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:04.194 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:06:10.184 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:04.234 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:06:10.224 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:04.246 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:06:10.237 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:04.247 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:06:10.214 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:04.251 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:06:10.218 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:04.257 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:06:10.247 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:04.288 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:06:10.255 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:04.300 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:06:10.291 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:07.007 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:07.126 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.125+0000 7f47908a8640 10 monclient: get_monmap_and_config 2024-03-25T18:14:07.127 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.125+0000 7f47908a8640 10 monclient: build_initial_monmap 2024-03-25T18:14:07.127 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.125+0000 7f47908a8640 10 monclient: monmap: 2024-03-25T18:14:07.127 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:14:07.127 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:07.127 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:14:07.126203+0000 2024-03-25T18:14:07.127 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:14:07.126203+0000 2024-03-25T18:14:07.127 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:07.127 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:07.127 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:14:07.127 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:14:07.127 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:14:07.127 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:07.127 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.125+0000 7f47908a8640 1 Processor -- start 2024-03-25T18:14:07.128 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.125+0000 7f47908a8640 1 -- start start 2024-03-25T18:14:07.128 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.125+0000 7f47908a8640 10 monclient: init 2024-03-25T18:14:07.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.126+0000 7f47908a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:07.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.126+0000 7f47908a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:07.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.126+0000 7f47908a8640 10 monclient(hunting): picked mon.noname-c con 0x7f478c0a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:14:07.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.126+0000 7f47908a8640 10 monclient(hunting): picked mon.noname-b con 0x7f478c13f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:14:07.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.126+0000 7f47908a8640 10 monclient(hunting): picked mon.noname-a con 0x7f478c1430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:14:07.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.126+0000 7f47908a8640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f478c142520 con 0x7f478c1430c0 2024-03-25T18:14:07.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.126+0000 7f47908a8640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f478c0b28a0 con 0x7f478c0a98f0 2024-03-25T18:14:07.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.126+0000 7f47908a8640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f478c0b2a30 con 0x7f478c13f710 2024-03-25T18:14:07.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.126+0000 7f47908a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:07.130 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.126+0000 7f47908a8640 10 monclient(hunting): authenticate will time out at 7328.437655s 2024-03-25T18:14:07.136 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.135+0000 7f478ad9d640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f478c1430c0 0x7f478c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:47358/0 (socket says 172.21.15.156:47358) 2024-03-25T18:14:07.136 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.136+0000 7f478ad9d640 1 -- 172.21.15.156:0/2416907392 learned_addr learned my addr 172.21.15.156:0/2416907392 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:14:07.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.143+0000 7f478959a640 1 -- 172.21.15.156:0/2416907392 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2890287387 0 0) 0x7f478c142520 con 0x7f478c1430c0 2024-03-25T18:14:07.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.143+0000 7f478959a640 10 monclient(hunting): _negotiate 2024-03-25T18:14:07.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.143+0000 7f478959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:07.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.143+0000 7f478959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:07.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.143+0000 7f478959a640 10 monclient(hunting): my global_id is 4482 2024-03-25T18:14:07.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.143+0000 7f478959a640 1 -- 172.21.15.156:0/2416907392 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4774003520 con 0x7f478c1430c0 2024-03-25T18:14:07.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.152+0000 7f478959a640 1 -- 172.21.15.156:0/2416907392 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623215484 0 0) 0x7f478c0b2a30 con 0x7f478c13f710 2024-03-25T18:14:07.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.152+0000 7f478959a640 10 monclient(hunting): _negotiate 2024-03-25T18:14:07.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.152+0000 7f478959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:07.154 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.152+0000 7f478959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:07.154 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.152+0000 7f478959a640 10 monclient(hunting): my global_id is 4490 2024-03-25T18:14:07.154 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.152+0000 7f478959a640 1 -- 172.21.15.156:0/2416907392 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f478c142520 con 0x7f478c13f710 2024-03-25T18:14:07.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.156+0000 7f478959a640 1 -- 172.21.15.156:0/2416907392 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 649921777 0 0) 0x7f4774003520 con 0x7f478c1430c0 2024-03-25T18:14:07.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.156+0000 7f478959a640 1 -- 172.21.15.156:0/2416907392 >> v1:172.21.15.156:6790/0 conn(0x7f478c13f710 legacy=0x7f478c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.156+0000 7f478959a640 1 -- 172.21.15.156:0/2416907392 >> v1:172.21.15.194:6789/0 conn(0x7f478c0a98f0 legacy=0x7f478c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.156+0000 7f478959a640 10 monclient: _finish_hunting 0 2024-03-25T18:14:07.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.156+0000 7f478959a640 1 monclient: found mon.noname-a 2024-03-25T18:14:07.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.156+0000 7f478959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:07.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.156+0000 7f478959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:14:07.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.156+0000 7f478959a640 1 -- 172.21.15.156:0/2416907392 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f478c0b44c0 con 0x7f478c1430c0 2024-03-25T18:14:07.159 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.157+0000 7f478959a640 10 monclient: _finish_auth 0 2024-03-25T18:14:07.159 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.157+0000 7f478959a640 10 monclient: _check_auth_tickets 2024-03-25T18:14:07.159 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.157+0000 7f478959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:07.159 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.157+0000 7f478959a640 1 -- 172.21.15.156:0/2416907392 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f4784001a10 con 0x7f478c1430c0 2024-03-25T18:14:07.159 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.157+0000 7f47908a8640 5 monclient: authenticate success, global_id 4482 2024-03-25T18:14:07.159 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.157+0000 7f47908a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:07.159 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.157+0000 7f478959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:07.159 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.157+0000 7f478959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:07.159 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.157+0000 7f478959a640 10 monclient: dump: 2024-03-25T18:14:07.159 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:07.159 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:07.159 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:07.159 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:07.159 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:07.160 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:07.160 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:07.160 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:07.160 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:07.160 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:07.160 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.157+0000 7f47908a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:07.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.171+0000 7f478959a640 1 -- 172.21.15.156:0/2416907392 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4784001b70 con 0x7f478c1430c0 2024-03-25T18:14:07.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.171+0000 7f478959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:07.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.171+0000 7f478959a640 1 -- 172.21.15.156:0/2416907392 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f4784001f80 con 0x7f478c1430c0 2024-03-25T18:14:07.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.171+0000 7f478959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:07.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.171+0000 7f478959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:07.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.171+0000 7f478959a640 10 monclient: dump: 2024-03-25T18:14:07.172 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:07.172 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:07.172 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:07.173 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:07.173 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:07.173 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:07.173 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:07.173 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:07.173 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:07.174 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:07.174 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.171+0000 7f47908a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:07.174 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.171+0000 7f47908a8640 4 set_mon_vals no callback set 2024-03-25T18:14:07.176 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.172+0000 7f47908a8640 10 monclient: shutdown 2024-03-25T18:14:07.176 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.172+0000 7f47908a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:07.176 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.172+0000 7f47908a8640 1 -- 172.21.15.156:0/2416907392 >> v1:172.21.15.156:6789/0 conn(0x7f478c1430c0 legacy=0x7f478c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.176 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.172+0000 7f47908a8640 1 -- 172.21.15.156:0/2416907392 shutdown_connections 2024-03-25T18:14:07.176 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.172+0000 7f47908a8640 1 -- 172.21.15.156:0/2416907392 >> 172.21.15.156:0/2416907392 conn(0x7f478c0b7ea0 msgr2=0x7f478c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:07.176 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.172+0000 7f47908a8640 1 -- 172.21.15.156:0/2416907392 shutdown_connections 2024-03-25T18:14:07.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.172+0000 7f47908a8640 1 -- 172.21.15.156:0/2416907392 wait complete. 2024-03-25T18:14:07.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.173+0000 7f47908a8640 10 monclient: build_initial_monmap 2024-03-25T18:14:07.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.173+0000 7f47908a8640 10 monclient: monmap: 2024-03-25T18:14:07.177 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:14:07.177 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:07.177 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:14:07.177 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:14:07.177 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:07.177 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:14:07.177 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:14:07.177 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:14:07.177 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:14:07.177 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:07.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.173+0000 7f47908a8640 1 librados: starting msgr at 2024-03-25T18:14:07.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.173+0000 7f47908a8640 1 librados: starting objecter 2024-03-25T18:14:07.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.174+0000 7f47908a8640 1 Processor -- start 2024-03-25T18:14:07.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.174+0000 7f47908a8640 1 -- start start 2024-03-25T18:14:07.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.174+0000 7f47908a8640 1 librados: setting wanted keys 2024-03-25T18:14:07.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.174+0000 7f47908a8640 1 librados: calling monclient init 2024-03-25T18:14:07.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.174+0000 7f47908a8640 10 monclient: init 2024-03-25T18:14:07.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.174+0000 7f47908a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:07.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.174+0000 7f47908a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:07.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.174+0000 7f47908a8640 10 monclient(hunting): picked mon.noname-a con 0x7f478c0a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:14:07.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.174+0000 7f47908a8640 10 monclient(hunting): picked mon.noname-c con 0x7f478c13f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:14:07.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.174+0000 7f47908a8640 10 monclient(hunting): picked mon.noname-b con 0x7f478c1430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:14:07.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.174+0000 7f47908a8640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f478c136e20 con 0x7f478c0a98f0 2024-03-25T18:14:07.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.174+0000 7f47908a8640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f478c0b7940 con 0x7f478c1430c0 2024-03-25T18:14:07.179 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.174+0000 7f47908a8640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f478c1351a0 con 0x7f478c13f710 2024-03-25T18:14:07.179 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.174+0000 7f47908a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:07.179 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.174+0000 7f47908a8640 10 monclient(hunting): authenticate will time out at 7328.485679s 2024-03-25T18:14:07.189 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.188+0000 7f4789d9b640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f478c13f710 0x7f478c1db1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:60084/0 (socket says 172.21.15.156:60084) 2024-03-25T18:14:07.189 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.188+0000 7f4789d9b640 1 -- 172.21.15.156:0/2405798693 learned_addr learned my addr 172.21.15.156:0/2405798693 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:14:07.191 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.190+0000 7f477affd640 10 client.?.objecter ms_handle_connect 0x7f478c0a98f0 2024-03-25T18:14:07.191 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.190+0000 7f477affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:07.191 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.190+0000 7f477affd640 10 client.?.objecter ms_handle_connect 0x7f478c13f710 2024-03-25T18:14:07.191 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.190+0000 7f477affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:07.196 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.195+0000 7f477affd640 10 client.?.objecter ms_handle_connect 0x7f478c1430c0 2024-03-25T18:14:07.196 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.195+0000 7f477affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:07.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.202+0000 7f477affd640 1 -- 172.21.15.156:0/2405798693 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3291974637 0 0) 0x7f478c136e20 con 0x7f478c0a98f0 2024-03-25T18:14:07.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.202+0000 7f477affd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:07.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.202+0000 7f477affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:07.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.202+0000 7f477affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:07.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.202+0000 7f477affd640 10 monclient(hunting): my global_id is 4485 2024-03-25T18:14:07.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.202+0000 7f477affd640 1 -- 172.21.15.156:0/2405798693 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f476c004eb0 con 0x7f478c0a98f0 2024-03-25T18:14:07.206 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.205+0000 7f477affd640 1 -- 172.21.15.156:0/2405798693 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3574473846 0 0) 0x7f478c1351a0 con 0x7f478c13f710 2024-03-25T18:14:07.206 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.205+0000 7f477affd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:07.206 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.205+0000 7f477affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:07.206 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.205+0000 7f477affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:07.206 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.205+0000 7f477affd640 10 monclient(hunting): my global_id is 4493 2024-03-25T18:14:07.207 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.205+0000 7f477affd640 1 -- 172.21.15.156:0/2405798693 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f478c136e20 con 0x7f478c13f710 2024-03-25T18:14:07.219 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.217+0000 7f477affd640 1 -- 172.21.15.156:0/2405798693 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870773094 0 0) 0x7f478c0b7940 con 0x7f478c1430c0 2024-03-25T18:14:07.219 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.217+0000 7f477affd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:07.219 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.217+0000 7f477affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:07.219 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.217+0000 7f477affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:07.219 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.217+0000 7f477affd640 10 monclient(hunting): my global_id is 4489 2024-03-25T18:14:07.219 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.217+0000 7f477affd640 1 -- 172.21.15.156:0/2405798693 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f478c1351a0 con 0x7f478c1430c0 2024-03-25T18:14:07.219 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.217+0000 7f477affd640 1 -- 172.21.15.156:0/2405798693 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3056446496 0 0) 0x7f476c004eb0 con 0x7f478c0a98f0 2024-03-25T18:14:07.219 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.217+0000 7f477affd640 1 -- 172.21.15.156:0/2405798693 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f478c0b7940 con 0x7f478c0a98f0 2024-03-25T18:14:07.219 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.217+0000 7f477affd640 1 -- 172.21.15.156:0/2405798693 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f477c0018c0 con 0x7f478c0a98f0 2024-03-25T18:14:07.219 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.217+0000 7f477affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:07.219 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.217+0000 7f477affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:07.220 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.217+0000 7f477affd640 10 monclient(hunting): dump: 2024-03-25T18:14:07.220 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:07.220 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:07.220 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:07.220 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:07.220 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:07.220 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:07.220 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:07.220 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:07.221 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:07.221 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:07.221 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.217+0000 7f477affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:07.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.229+0000 7f477affd640 1 -- 172.21.15.156:0/2405798693 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3980507621 0 0) 0x7f478c1351a0 con 0x7f478c1430c0 2024-03-25T18:14:07.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.229+0000 7f477affd640 1 -- 172.21.15.156:0/2405798693 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f476c004eb0 con 0x7f478c1430c0 2024-03-25T18:14:07.234 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.233+0000 7f477affd640 1 -- 172.21.15.156:0/2405798693 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f4784001e80 con 0x7f478c1430c0 2024-03-25T18:14:07.234 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.233+0000 7f477affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:07.234 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.233+0000 7f477affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:07.234 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.233+0000 7f477affd640 10 monclient(hunting): dump: 2024-03-25T18:14:07.234 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:07.235 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:07.235 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:07.235 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:07.235 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:07.235 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:07.235 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:07.235 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:07.235 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:07.235 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:07.235 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.233+0000 7f477affd640 1 -- 172.21.15.156:0/2405798693 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3473217438 0 0) 0x7f478c136e20 con 0x7f478c13f710 2024-03-25T18:14:07.235 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.233+0000 7f477affd640 1 -- 172.21.15.156:0/2405798693 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f478c1351a0 con 0x7f478c13f710 2024-03-25T18:14:07.235 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.233+0000 7f477affd640 1 -- 172.21.15.156:0/2405798693 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f47800019f0 con 0x7f478c13f710 2024-03-25T18:14:07.235 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.233+0000 7f477affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:07.235 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.233+0000 7f477affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:07.236 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.233+0000 7f477affd640 10 monclient(hunting): dump: 2024-03-25T18:14:07.236 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:07.236 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:07.236 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:07.236 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:07.236 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:07.236 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:07.236 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:07.236 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:07.236 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:07.236 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:07.238 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.234+0000 7f477affd640 1 -- 172.21.15.156:0/2405798693 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3879052969 0 0) 0x7f478c0b7940 con 0x7f478c0a98f0 2024-03-25T18:14:07.238 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.234+0000 7f477affd640 1 -- 172.21.15.156:0/2405798693 >> v1:172.21.15.156:6790/0 conn(0x7f478c13f710 legacy=0x7f478c1db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.238 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.234+0000 7f477affd640 1 -- 172.21.15.156:0/2405798693 >> v1:172.21.15.194:6789/0 conn(0x7f478c1430c0 legacy=0x7f478c1e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.238 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.234+0000 7f477affd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:07.238 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.234+0000 7f477affd640 1 monclient: found mon.a 2024-03-25T18:14:07.238 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.234+0000 7f477affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:07.238 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.234+0000 7f477affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:14:07.239 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.234+0000 7f477affd640 1 -- 172.21.15.156:0/2405798693 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f478c135330 con 0x7f478c0a98f0 2024-03-25T18:14:07.239 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.234+0000 7f477affd640 10 monclient: _finish_auth 0 2024-03-25T18:14:07.239 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.234+0000 7f477affd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:07.239 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.234+0000 7f477affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:07.239 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.234+0000 7f47908a8640 5 monclient: authenticate success, global_id 4485 2024-03-25T18:14:07.239 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.234+0000 7f47908a8640 10 monclient: _renew_subs 2024-03-25T18:14:07.239 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.234+0000 7f47908a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:14:07.239 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.234+0000 7f47908a8640 1 -- 172.21.15.156:0/2405798693 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f478c135560 con 0x7f478c0a98f0 2024-03-25T18:14:07.239 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.235+0000 7f47908a8640 10 client.4485.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:07.239 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.235+0000 7f47908a8640 10 monclient: _renew_subs 2024-03-25T18:14:07.239 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.235+0000 7f47908a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:14:07.239 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.235+0000 7f47908a8640 1 -- 172.21.15.156:0/2405798693 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f478c13b490 con 0x7f478c0a98f0 2024-03-25T18:14:07.239 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.235+0000 7f47908a8640 1 librados: init done 2024-03-25T18:14:07.239 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.236+0000 7f47908a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:07.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.236+0000 7f47908a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:07.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.236+0000 7f47908a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:14:07.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.236+0000 7f47908a8640 1 -- 172.21.15.156:0/2405798693 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f478c0b7ad0 con 0x7f478c0a98f0 2024-03-25T18:14:07.541 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.540+0000 7f477affd640 1 -- 172.21.15.156:0/2405798693 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f477c0035f0 con 0x7f478c0a98f0 2024-03-25T18:14:07.541 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.540+0000 7f477affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:07.541 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.540+0000 7f477affd640 1 -- 172.21.15.156:0/2405798693 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f477c003900 con 0x7f478c0a98f0 2024-03-25T18:14:07.541 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.540+0000 7f477affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:07.541 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.540+0000 7f477affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:07.542 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.540+0000 7f477affd640 10 monclient: dump: 2024-03-25T18:14:07.542 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:07.542 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:07.542 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:07.542 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:07.542 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:07.542 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:07.542 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:07.542 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:07.542 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:07.542 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:07.542 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.540+0000 7f4788d99640 4 set_mon_vals no callback set 2024-03-25T18:14:07.543 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.541+0000 7f477affd640 1 -- 172.21.15.156:0/2405798693 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f477c0198d0 con 0x7f478c0a98f0 2024-03-25T18:14:07.543 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.541+0000 7f4789d9b640 10 monclient: get_auth_request con 0x7f476c06e080 auth_method 0 2024-03-25T18:14:07.543 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.542+0000 7f477affd640 10 client.4485.objecter ms_handle_connect 0x7f476c06e080 2024-03-25T18:14:07.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.546+0000 7f477affd640 1 -- 172.21.15.156:0/2405798693 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f477c053540 con 0x7f478c0a98f0 2024-03-25T18:14:07.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.546+0000 7f477affd640 10 client.4485.objecter ms_dispatch 0x7f478c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:07.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.546+0000 7f477affd640 3 client.4485.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:07.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.546+0000 7f477affd640 3 client.4485.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:07.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.546+0000 7f477affd640 20 client.4485.objecter dump_active .. 0 homeless 2024-03-25T18:14:07.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.546+0000 7f477affd640 1 -- 172.21.15.156:0/2405798693 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f477c04ee80 con 0x7f478c0a98f0 2024-03-25T18:14:07.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.546+0000 7f477affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:07.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.546+0000 7f477affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:07.791 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.790+0000 7f47908a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:07.791 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.790+0000 7f47908a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:07.792 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.790+0000 7f47908a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:14:07.792 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.790+0000 7f47908a8640 1 -- 172.21.15.156:0/2405798693 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f478c0b7730 con 0x7f478c0a98f0 2024-03-25T18:14:07.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.845+0000 7f477affd640 1 -- 172.21.15.156:0/2405798693 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f477c056ba0 con 0x7f478c0a98f0 2024-03-25T18:14:07.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.845+0000 7f477affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:07.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.845+0000 7f477affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:07.846 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:14:07.846 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:07.849 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.847+0000 7f47908a8640 10 librados: watch_flush enter 2024-03-25T18:14:07.849 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.847+0000 7f47908a8640 10 librados: watch_flush exit 2024-03-25T18:14:07.849 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.847+0000 7f47908a8640 20 client.4485.objecter shutdown clearing up homeless session... 2024-03-25T18:14:07.849 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.847+0000 7f47908a8640 10 client.4485.objecter successfully canceled tick 2024-03-25T18:14:07.849 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.847+0000 7f47908a8640 1 -- 172.21.15.156:0/2405798693 >> v1:172.21.15.194:6800/2946427713 conn(0x7f476c06e080 legacy=0x7f476c070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.849 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.847+0000 7f47908a8640 10 monclient: shutdown 2024-03-25T18:14:07.849 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.847+0000 7f47908a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:07.849 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.847+0000 7f47908a8640 1 -- 172.21.15.156:0/2405798693 >> v1:172.21.15.156:6789/0 conn(0x7f478c0a98f0 legacy=0x7f478c1daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.849 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.847+0000 7f47908a8640 1 -- 172.21.15.156:0/2405798693 shutdown_connections 2024-03-25T18:14:07.849 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.847+0000 7f47908a8640 1 -- 172.21.15.156:0/2405798693 >> 172.21.15.156:0/2405798693 conn(0x7f478c0b7ea0 msgr2=0x7f478c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:07.849 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.847+0000 7f47908a8640 1 -- 172.21.15.156:0/2405798693 shutdown_connections 2024-03-25T18:14:07.849 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.847+0000 7f47908a8640 1 -- 172.21.15.156:0/2405798693 wait complete. 2024-03-25T18:14:07.849 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:07.848+0000 7f47908a8640 1 librados: shutdown 2024-03-25T18:14:07.861 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:14.668 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:14.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.782+0000 7eff8c5cf640 10 monclient: get_monmap_and_config 2024-03-25T18:14:14.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.782+0000 7eff8c5cf640 10 monclient: build_initial_monmap 2024-03-25T18:14:14.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.782+0000 7eff8c5cf640 10 monclient: monmap: 2024-03-25T18:14:14.783 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:14:14.784 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:14.784 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:14:14.783396+0000 2024-03-25T18:14:14.784 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:14:14.783396+0000 2024-03-25T18:14:14.784 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:14.784 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:14.784 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:14:14.784 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:14:14.784 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:14:14.784 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:14.785 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.783+0000 7eff8c5cf640 1 Processor -- start 2024-03-25T18:14:14.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.783+0000 7eff8c5cf640 1 -- start start 2024-03-25T18:14:14.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.783+0000 7eff8c5cf640 10 monclient: init 2024-03-25T18:14:14.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.784+0000 7eff8c5cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:14.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.784+0000 7eff8c5cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:14.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.784+0000 7eff8c5cf640 10 monclient(hunting): picked mon.noname-a con 0x7eff840a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:14:14.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.784+0000 7eff8c5cf640 10 monclient(hunting): picked mon.noname-b con 0x7eff8413f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:14:14.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.784+0000 7eff8c5cf640 10 monclient(hunting): picked mon.noname-c con 0x7eff841430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:14:14.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.784+0000 7eff8c5cf640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff84142520 con 0x7eff840a98f0 2024-03-25T18:14:14.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.784+0000 7eff8c5cf640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff840b28a0 con 0x7eff841430c0 2024-03-25T18:14:14.786 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.784+0000 7eff8c5cf640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff840b2a30 con 0x7eff8413f710 2024-03-25T18:14:14.787 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.784+0000 7eff8c5cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:14.787 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.784+0000 7eff8c5cf640 10 monclient(hunting): authenticate will time out at 7336.095303s 2024-03-25T18:14:14.802 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.801+0000 7eff8ab6c640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7eff841430c0 0x7eff8414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:50364/0 (socket says 172.21.15.156:50364) 2024-03-25T18:14:14.802 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.801+0000 7eff8ab6c640 1 -- 172.21.15.156:0/1893011598 learned_addr learned my addr 172.21.15.156:0/1893011598 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:14:14.808 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.807+0000 7eff89369640 1 -- 172.21.15.156:0/1893011598 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 226697229 0 0) 0x7eff840b2a30 con 0x7eff8413f710 2024-03-25T18:14:14.808 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.807+0000 7eff89369640 10 monclient(hunting): _negotiate 2024-03-25T18:14:14.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.807+0000 7eff89369640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:14.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.807+0000 7eff89369640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:14.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.807+0000 7eff89369640 10 monclient(hunting): my global_id is 4496 2024-03-25T18:14:14.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.807+0000 7eff89369640 1 -- 172.21.15.156:0/1893011598 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff70003520 con 0x7eff8413f710 2024-03-25T18:14:14.811 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.810+0000 7eff89369640 1 -- 172.21.15.156:0/1893011598 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983961402 0 0) 0x7eff840b28a0 con 0x7eff841430c0 2024-03-25T18:14:14.811 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.810+0000 7eff89369640 10 monclient(hunting): _negotiate 2024-03-25T18:14:14.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.810+0000 7eff89369640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:14.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.810+0000 7eff89369640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:14.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.810+0000 7eff89369640 10 monclient(hunting): my global_id is 4492 2024-03-25T18:14:14.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.810+0000 7eff89369640 1 -- 172.21.15.156:0/1893011598 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff840b2a30 con 0x7eff841430c0 2024-03-25T18:14:14.817 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.816+0000 7eff89369640 1 -- 172.21.15.156:0/1893011598 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4049627866 0 0) 0x7eff840b2a30 con 0x7eff841430c0 2024-03-25T18:14:14.817 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.816+0000 7eff89369640 1 -- 172.21.15.156:0/1893011598 >> v1:172.21.15.156:6790/0 conn(0x7eff8413f710 legacy=0x7eff841429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:14.817 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.816+0000 7eff89369640 1 -- 172.21.15.156:0/1893011598 >> v1:172.21.15.156:6789/0 conn(0x7eff840a98f0 legacy=0x7eff840a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:14.818 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.816+0000 7eff89369640 10 monclient: _finish_hunting 0 2024-03-25T18:14:14.818 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.816+0000 7eff89369640 1 monclient: found mon.noname-c 2024-03-25T18:14:14.818 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.816+0000 7eff89369640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:14.818 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.816+0000 7eff89369640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:14:14.818 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.816+0000 7eff89369640 1 -- 172.21.15.156:0/1893011598 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff840b44c0 con 0x7eff841430c0 2024-03-25T18:14:14.818 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.816+0000 7eff89369640 10 monclient: _finish_auth 0 2024-03-25T18:14:14.818 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.816+0000 7eff89369640 10 monclient: _check_auth_tickets 2024-03-25T18:14:14.818 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.816+0000 7eff89369640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:14.818 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.816+0000 7eff89369640 1 -- 172.21.15.156:0/1893011598 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7eff7c0019f0 con 0x7eff841430c0 2024-03-25T18:14:14.818 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.816+0000 7eff8c5cf640 5 monclient: authenticate success, global_id 4492 2024-03-25T18:14:14.818 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.816+0000 7eff8c5cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:14.818 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.816+0000 7eff89369640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:14.818 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.816+0000 7eff89369640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:14.818 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.816+0000 7eff89369640 10 monclient: dump: 2024-03-25T18:14:14.819 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:14.819 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:14.819 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:14.819 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:14.819 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:14.819 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:14.819 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:14.819 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:14.819 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:14.819 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:14.819 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.816+0000 7eff8c5cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:14.828 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.827+0000 7eff89369640 1 -- 172.21.15.156:0/1893011598 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff7c001b50 con 0x7eff841430c0 2024-03-25T18:14:14.829 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.827+0000 7eff89369640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:14.829 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.827+0000 7eff89369640 1 -- 172.21.15.156:0/1893011598 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7eff7c001f20 con 0x7eff841430c0 2024-03-25T18:14:14.829 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.827+0000 7eff89369640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:14.829 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.827+0000 7eff89369640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:14.829 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.827+0000 7eff89369640 10 monclient: dump: 2024-03-25T18:14:14.829 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:14.829 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:14.829 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:14.829 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:14.829 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:14.829 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:14.829 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:14.829 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:14.829 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:14.829 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:14.829 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.827+0000 7eff8c5cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:14.830 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.827+0000 7eff8c5cf640 4 set_mon_vals no callback set 2024-03-25T18:14:14.831 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.828+0000 7eff8c5cf640 10 monclient: shutdown 2024-03-25T18:14:14.831 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.828+0000 7eff8c5cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:14.831 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.828+0000 7eff8c5cf640 1 -- 172.21.15.156:0/1893011598 >> v1:172.21.15.194:6789/0 conn(0x7eff841430c0 legacy=0x7eff8414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:14.831 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.828+0000 7eff8c5cf640 1 -- 172.21.15.156:0/1893011598 shutdown_connections 2024-03-25T18:14:14.831 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.828+0000 7eff8c5cf640 1 -- 172.21.15.156:0/1893011598 >> 172.21.15.156:0/1893011598 conn(0x7eff840b7ea0 msgr2=0x7eff840b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:14.831 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.828+0000 7eff8c5cf640 1 -- 172.21.15.156:0/1893011598 shutdown_connections 2024-03-25T18:14:14.831 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.828+0000 7eff8c5cf640 1 -- 172.21.15.156:0/1893011598 wait complete. 2024-03-25T18:14:14.831 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.829+0000 7eff8c5cf640 10 monclient: build_initial_monmap 2024-03-25T18:14:14.831 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.829+0000 7eff8c5cf640 10 monclient: monmap: 2024-03-25T18:14:14.831 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:14:14.832 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:14.832 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:14:14.832 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:14:14.832 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:14.832 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:14:14.832 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:14:14.832 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:14:14.832 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:14:14.832 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:14.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.830+0000 7eff8c5cf640 1 librados: starting msgr at 2024-03-25T18:14:14.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.830+0000 7eff8c5cf640 1 librados: starting objecter 2024-03-25T18:14:14.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.830+0000 7eff8c5cf640 1 Processor -- start 2024-03-25T18:14:14.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.830+0000 7eff8c5cf640 1 -- start start 2024-03-25T18:14:14.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.830+0000 7eff8c5cf640 1 librados: setting wanted keys 2024-03-25T18:14:14.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.830+0000 7eff8c5cf640 1 librados: calling monclient init 2024-03-25T18:14:14.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.830+0000 7eff8c5cf640 10 monclient: init 2024-03-25T18:14:14.833 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.831+0000 7eff8c5cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:14.834 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.831+0000 7eff8c5cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:14.834 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.831+0000 7eff8c5cf640 10 monclient(hunting): picked mon.noname-c con 0x7eff840a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:14:14.834 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.831+0000 7eff8c5cf640 10 monclient(hunting): picked mon.noname-b con 0x7eff8413f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:14:14.834 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.831+0000 7eff8c5cf640 10 monclient(hunting): picked mon.noname-a con 0x7eff841430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:14:14.834 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.831+0000 7eff8c5cf640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff84137e20 con 0x7eff841430c0 2024-03-25T18:14:14.834 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.831+0000 7eff8c5cf640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff840b1720 con 0x7eff8413f710 2024-03-25T18:14:14.834 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.831+0000 7eff8c5cf640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff84138710 con 0x7eff840a98f0 2024-03-25T18:14:14.834 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.831+0000 7eff8c5cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:14.834 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.831+0000 7eff8c5cf640 10 monclient(hunting): authenticate will time out at 7336.142183s 2024-03-25T18:14:14.839 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.839+0000 7eff8ab6c640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7eff841430c0 0x7eff841daf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:54432/0 (socket says 172.21.15.156:54432) 2024-03-25T18:14:14.840 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.839+0000 7eff8ab6c640 1 -- 172.21.15.156:0/2392438903 learned_addr learned my addr 172.21.15.156:0/2392438903 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:14:14.841 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.840+0000 7eff7affd640 10 client.?.objecter ms_handle_connect 0x7eff841430c0 2024-03-25T18:14:14.841 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.840+0000 7eff7affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:14.844 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.844+0000 7eff7affd640 10 client.?.objecter ms_handle_connect 0x7eff840a98f0 2024-03-25T18:14:14.845 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.844+0000 7eff7affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:14.848 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.847+0000 7eff7affd640 1 -- 172.21.15.156:0/2392438903 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2069408791 0 0) 0x7eff84137e20 con 0x7eff841430c0 2024-03-25T18:14:14.849 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.848+0000 7eff7affd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:14.849 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.848+0000 7eff7affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:14.849 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.848+0000 7eff7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:14.849 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.848+0000 7eff7affd640 10 monclient(hunting): my global_id is 4491 2024-03-25T18:14:14.849 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.848+0000 7eff7affd640 1 -- 172.21.15.156:0/2392438903 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff68004eb0 con 0x7eff841430c0 2024-03-25T18:14:14.857 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.856+0000 7eff7affd640 1 -- 172.21.15.156:0/2392438903 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2766653631 0 0) 0x7eff84138710 con 0x7eff840a98f0 2024-03-25T18:14:14.857 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.856+0000 7eff7affd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:14.857 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.856+0000 7eff7affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:14.857 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.856+0000 7eff7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:14.857 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.856+0000 7eff7affd640 10 monclient(hunting): my global_id is 4499 2024-03-25T18:14:14.857 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.856+0000 7eff7affd640 1 -- 172.21.15.156:0/2392438903 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff84137e20 con 0x7eff840a98f0 2024-03-25T18:14:14.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.857+0000 7eff7affd640 1 -- 172.21.15.156:0/2392438903 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2680155123 0 0) 0x7eff68004eb0 con 0x7eff841430c0 2024-03-25T18:14:14.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.857+0000 7eff7affd640 1 -- 172.21.15.156:0/2392438903 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff84138710 con 0x7eff841430c0 2024-03-25T18:14:14.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.862+0000 7eff7affd640 10 client.?.objecter ms_handle_connect 0x7eff8413f710 2024-03-25T18:14:14.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.862+0000 7eff7affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:14.866 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.864+0000 7eff7affd640 1 -- 172.21.15.156:0/2392438903 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7eff7c001fc0 con 0x7eff841430c0 2024-03-25T18:14:14.866 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.864+0000 7eff7affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:14.866 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.864+0000 7eff7affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:14.866 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.864+0000 7eff7affd640 10 monclient(hunting): dump: 2024-03-25T18:14:14.866 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:14.866 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:14.866 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:14.866 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:14.867 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:14.867 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:14.867 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:14.867 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:14.867 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:14.867 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:14.867 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.864+0000 7eff7affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:14.867 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.864+0000 7eff7affd640 1 -- 172.21.15.156:0/2392438903 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3684255385 0 0) 0x7eff84138710 con 0x7eff841430c0 2024-03-25T18:14:14.867 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.865+0000 7eff7affd640 1 -- 172.21.15.156:0/2392438903 >> v1:172.21.15.156:6790/0 conn(0x7eff840a98f0 legacy=0x7eff84137710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:14.867 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.865+0000 7eff7affd640 1 -- 172.21.15.156:0/2392438903 >> v1:172.21.15.194:6789/0 conn(0x7eff8413f710 legacy=0x7eff841da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:14.867 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.865+0000 7eff7affd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:14.867 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.865+0000 7eff7affd640 1 monclient: found mon.a 2024-03-25T18:14:14.867 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.865+0000 7eff7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:14.867 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.865+0000 7eff7affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:14:14.867 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.865+0000 7eff7affd640 1 -- 172.21.15.156:0/2392438903 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff84138040 con 0x7eff841430c0 2024-03-25T18:14:14.868 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.865+0000 7eff7affd640 10 monclient: _finish_auth 0 2024-03-25T18:14:14.868 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.865+0000 7eff7affd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:14.868 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.865+0000 7eff7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:14.868 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.865+0000 7eff8c5cf640 5 monclient: authenticate success, global_id 4491 2024-03-25T18:14:14.868 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.865+0000 7eff8c5cf640 10 monclient: _renew_subs 2024-03-25T18:14:14.868 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.865+0000 7eff8c5cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:14:14.868 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.865+0000 7eff8c5cf640 1 -- 172.21.15.156:0/2392438903 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff841382f0 con 0x7eff841430c0 2024-03-25T18:14:14.868 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.865+0000 7eff8c5cf640 10 client.4491.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:14.868 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.865+0000 7eff8c5cf640 10 monclient: _renew_subs 2024-03-25T18:14:14.868 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.865+0000 7eff8c5cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:14:14.868 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.865+0000 7eff8c5cf640 1 -- 172.21.15.156:0/2392438903 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7eff841e2fb0 con 0x7eff841430c0 2024-03-25T18:14:14.868 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.865+0000 7eff8c5cf640 1 librados: init done 2024-03-25T18:14:14.869 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.866+0000 7eff8c5cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:14.869 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.866+0000 7eff8c5cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:14.869 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.866+0000 7eff8c5cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:14:14.869 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:14.866+0000 7eff8c5cf640 1 -- 172.21.15.156:0/2392438903 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff840b3ef0 con 0x7eff841430c0 2024-03-25T18:14:15.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.161+0000 7eff7affd640 1 -- 172.21.15.156:0/2392438903 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff7c004e10 con 0x7eff841430c0 2024-03-25T18:14:15.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.161+0000 7eff7affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:15.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.161+0000 7eff7affd640 1 -- 172.21.15.156:0/2392438903 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7eff7c005120 con 0x7eff841430c0 2024-03-25T18:14:15.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.161+0000 7eff7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:15.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.161+0000 7eff7affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:15.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.161+0000 7eff7affd640 10 monclient: dump: 2024-03-25T18:14:15.163 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:15.163 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:15.164 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:15.164 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:15.164 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:15.164 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:15.164 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:15.164 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:15.164 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:15.164 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:15.164 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.161+0000 7eff88b68640 4 set_mon_vals no callback set 2024-03-25T18:14:15.165 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.162+0000 7eff7affd640 1 -- 172.21.15.156:0/2392438903 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7eff7c01b0f0 con 0x7eff841430c0 2024-03-25T18:14:15.165 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.163+0000 7eff8a36b640 10 monclient: get_auth_request con 0x7eff6806dd00 auth_method 0 2024-03-25T18:14:15.165 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.163+0000 7eff7affd640 10 client.4491.objecter ms_handle_connect 0x7eff6806dd00 2024-03-25T18:14:15.170 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.169+0000 7eff7affd640 1 -- 172.21.15.156:0/2392438903 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7eff7c054bf0 con 0x7eff841430c0 2024-03-25T18:14:15.170 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.169+0000 7eff7affd640 10 client.4491.objecter ms_dispatch 0x7eff84000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:15.170 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.169+0000 7eff7affd640 3 client.4491.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:15.170 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.169+0000 7eff7affd640 3 client.4491.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:15.170 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.169+0000 7eff7affd640 20 client.4491.objecter dump_active .. 0 homeless 2024-03-25T18:14:15.170 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.169+0000 7eff7affd640 1 -- 172.21.15.156:0/2392438903 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7eff7c050700 con 0x7eff841430c0 2024-03-25T18:14:15.170 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.169+0000 7eff7affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:15.171 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.169+0000 7eff7affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:15.431 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.430+0000 7eff8c5cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:15.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.430+0000 7eff8c5cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:15.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.430+0000 7eff8c5cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:14:15.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.430+0000 7eff8c5cf640 1 -- 172.21.15.156:0/2392438903 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff84138900 con 0x7eff841430c0 2024-03-25T18:14:15.467 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.466+0000 7eff7affd640 1 -- 172.21.15.156:0/2392438903 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7eff7c052630 con 0x7eff841430c0 2024-03-25T18:14:15.468 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.466+0000 7eff7affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:15.468 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.466+0000 7eff7affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:15.468 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:14:15.468 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:15.470 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.469+0000 7eff8c5cf640 10 librados: watch_flush enter 2024-03-25T18:14:15.471 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.469+0000 7eff8c5cf640 10 librados: watch_flush exit 2024-03-25T18:14:15.471 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.469+0000 7eff8c5cf640 20 client.4491.objecter shutdown clearing up homeless session... 2024-03-25T18:14:15.471 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.469+0000 7eff8c5cf640 10 client.4491.objecter successfully canceled tick 2024-03-25T18:14:15.471 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.469+0000 7eff8c5cf640 1 -- 172.21.15.156:0/2392438903 >> v1:172.21.15.194:6800/2946427713 conn(0x7eff6806dd00 legacy=0x7eff680701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.471 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.469+0000 7eff8c5cf640 10 monclient: shutdown 2024-03-25T18:14:15.471 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.469+0000 7eff8c5cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:15.471 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.469+0000 7eff8c5cf640 1 -- 172.21.15.156:0/2392438903 >> v1:172.21.15.156:6789/0 conn(0x7eff841430c0 legacy=0x7eff841daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.471 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.469+0000 7eff8c5cf640 1 -- 172.21.15.156:0/2392438903 shutdown_connections 2024-03-25T18:14:15.471 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.469+0000 7eff8c5cf640 1 -- 172.21.15.156:0/2392438903 >> 172.21.15.156:0/2392438903 conn(0x7eff840b7ea0 msgr2=0x7eff84074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:15.471 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.469+0000 7eff8c5cf640 1 -- 172.21.15.156:0/2392438903 shutdown_connections 2024-03-25T18:14:15.471 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.469+0000 7eff8c5cf640 1 -- 172.21.15.156:0/2392438903 wait complete. 2024-03-25T18:14:15.471 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:15.469+0000 7eff8c5cf640 1 librados: shutdown 2024-03-25T18:14:15.483 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:22.289 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:22.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.407+0000 7f3b72b77640 10 monclient: get_monmap_and_config 2024-03-25T18:14:22.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.407+0000 7f3b72b77640 10 monclient: build_initial_monmap 2024-03-25T18:14:22.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.407+0000 7f3b72b77640 10 monclient: monmap: 2024-03-25T18:14:22.409 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:14:22.409 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:22.409 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:14:22.408726+0000 2024-03-25T18:14:22.409 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:14:22.408726+0000 2024-03-25T18:14:22.409 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:22.409 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:22.409 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:14:22.409 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:14:22.409 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:14:22.409 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:22.410 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.408+0000 7f3b72b77640 1 Processor -- start 2024-03-25T18:14:22.410 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.408+0000 7f3b72b77640 1 -- start start 2024-03-25T18:14:22.410 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.408+0000 7f3b72b77640 10 monclient: init 2024-03-25T18:14:22.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.409+0000 7f3b72b77640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:22.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.409+0000 7f3b72b77640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:22.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.409+0000 7f3b72b77640 10 monclient(hunting): picked mon.noname-b con 0x7f3b6c13fc60 addr v1:172.21.15.156:6790/0 2024-03-25T18:14:22.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.409+0000 7f3b72b77640 10 monclient(hunting): picked mon.noname-a con 0x7f3b6c070570 addr v1:172.21.15.156:6789/0 2024-03-25T18:14:22.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.409+0000 7f3b72b77640 10 monclient(hunting): picked mon.noname-c con 0x7f3b6c0727a0 addr v1:172.21.15.194:6789/0 2024-03-25T18:14:22.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.409+0000 7f3b72b77640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b6c13c9f0 con 0x7f3b6c070570 2024-03-25T18:14:22.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.409+0000 7f3b72b77640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b6c06fac0 con 0x7f3b6c0727a0 2024-03-25T18:14:22.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.409+0000 7f3b72b77640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b6c0715d0 con 0x7f3b6c13fc60 2024-03-25T18:14:22.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.409+0000 7f3b72b77640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:22.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.409+0000 7f3b72b77640 10 monclient(hunting): authenticate will time out at 7343.720223s 2024-03-25T18:14:22.412 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.412+0000 7f3b70913640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f3b6c13fc60 0x7f3b6c06f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:51986/0 (socket says 172.21.15.156:51986) 2024-03-25T18:14:22.413 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.412+0000 7f3b70913640 1 -- 172.21.15.156:0/1976842400 learned_addr learned my addr 172.21.15.156:0/1976842400 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:14:22.466 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.465+0000 7f3b6b7fe640 1 -- 172.21.15.156:0/1976842400 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 468408333 0 0) 0x7f3b6c06fac0 con 0x7f3b6c0727a0 2024-03-25T18:14:22.466 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.465+0000 7f3b6b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:22.467 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.465+0000 7f3b6b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:22.467 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.465+0000 7f3b6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:22.467 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.465+0000 7f3b6b7fe640 10 monclient(hunting): my global_id is 4498 2024-03-25T18:14:22.467 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.465+0000 7f3b6b7fe640 1 -- 172.21.15.156:0/1976842400 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b5c003520 con 0x7f3b6c0727a0 2024-03-25T18:14:22.471 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.470+0000 7f3b6b7fe640 1 -- 172.21.15.156:0/1976842400 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3460976299 0 0) 0x7f3b6c0715d0 con 0x7f3b6c13fc60 2024-03-25T18:14:22.471 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.470+0000 7f3b6b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:22.471 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.470+0000 7f3b6b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:22.472 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.470+0000 7f3b6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:22.472 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.470+0000 7f3b6b7fe640 10 monclient(hunting): my global_id is 4502 2024-03-25T18:14:22.472 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.470+0000 7f3b6b7fe640 1 -- 172.21.15.156:0/1976842400 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b6c06fac0 con 0x7f3b6c13fc60 2024-03-25T18:14:22.484 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.483+0000 7f3b6b7fe640 1 -- 172.21.15.156:0/1976842400 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963008155 0 0) 0x7f3b6c13c9f0 con 0x7f3b6c070570 2024-03-25T18:14:22.484 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.483+0000 7f3b6b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:22.484 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.483+0000 7f3b6b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:22.485 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.483+0000 7f3b6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:22.485 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.483+0000 7f3b6b7fe640 10 monclient(hunting): my global_id is 4494 2024-03-25T18:14:22.485 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.483+0000 7f3b6b7fe640 1 -- 172.21.15.156:0/1976842400 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b6c0715d0 con 0x7f3b6c070570 2024-03-25T18:14:22.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.484+0000 7f3b6b7fe640 1 -- 172.21.15.156:0/1976842400 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 802180944 0 0) 0x7f3b5c003520 con 0x7f3b6c0727a0 2024-03-25T18:14:22.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.484+0000 7f3b6b7fe640 1 -- 172.21.15.156:0/1976842400 >> v1:172.21.15.156:6790/0 conn(0x7f3b6c13fc60 legacy=0x7f3b6c06f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:22.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.484+0000 7f3b6b7fe640 1 -- 172.21.15.156:0/1976842400 >> v1:172.21.15.156:6789/0 conn(0x7f3b6c070570 legacy=0x7f3b6c072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:22.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.484+0000 7f3b6b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:14:22.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.484+0000 7f3b6b7fe640 1 monclient: found mon.noname-c 2024-03-25T18:14:22.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.484+0000 7f3b6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:22.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.484+0000 7f3b6b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:14:22.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.484+0000 7f3b6b7fe640 1 -- 172.21.15.156:0/1976842400 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b6c0718b0 con 0x7f3b6c0727a0 2024-03-25T18:14:22.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.484+0000 7f3b6b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:14:22.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.484+0000 7f3b6b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:14:22.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.484+0000 7f3b6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:22.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.484+0000 7f3b6b7fe640 1 -- 172.21.15.156:0/1976842400 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f3b64001a10 con 0x7f3b6c0727a0 2024-03-25T18:14:22.486 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.484+0000 7f3b6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:22.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.484+0000 7f3b6b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:22.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.484+0000 7f3b6b7fe640 10 monclient: dump: 2024-03-25T18:14:22.487 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:22.487 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:22.487 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:22.487 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:22.487 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:22.487 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:22.487 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:22.487 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:22.487 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:22.487 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:22.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.484+0000 7f3b72b77640 5 monclient: authenticate success, global_id 4498 2024-03-25T18:14:22.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.484+0000 7f3b72b77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:22.503 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.502+0000 7f3b6b7fe640 1 -- 172.21.15.156:0/1976842400 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b64001b70 con 0x7f3b6c0727a0 2024-03-25T18:14:22.503 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.502+0000 7f3b6b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:22.503 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.502+0000 7f3b6b7fe640 1 -- 172.21.15.156:0/1976842400 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f3b64001f80 con 0x7f3b6c0727a0 2024-03-25T18:14:22.503 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.502+0000 7f3b6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:22.503 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.502+0000 7f3b6b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:22.504 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.502+0000 7f3b6b7fe640 10 monclient: dump: 2024-03-25T18:14:22.504 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:22.504 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:22.504 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:22.504 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:22.504 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:22.504 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:22.504 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:22.504 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:22.504 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:22.504 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:22.504 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.502+0000 7f3b72b77640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:22.504 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.502+0000 7f3b72b77640 4 set_mon_vals no callback set 2024-03-25T18:14:22.506 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.503+0000 7f3b72b77640 10 monclient: shutdown 2024-03-25T18:14:22.507 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.503+0000 7f3b72b77640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:22.507 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.503+0000 7f3b72b77640 1 -- 172.21.15.156:0/1976842400 >> v1:172.21.15.194:6789/0 conn(0x7f3b6c0727a0 legacy=0x7f3b6c14a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:22.507 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.503+0000 7f3b72b77640 1 -- 172.21.15.156:0/1976842400 shutdown_connections 2024-03-25T18:14:22.507 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.503+0000 7f3b72b77640 1 -- 172.21.15.156:0/1976842400 >> 172.21.15.156:0/1976842400 conn(0x7f3b6c0b4a80 msgr2=0x7f3b6c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:22.507 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.503+0000 7f3b72b77640 1 -- 172.21.15.156:0/1976842400 shutdown_connections 2024-03-25T18:14:22.507 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.503+0000 7f3b72b77640 1 -- 172.21.15.156:0/1976842400 wait complete. 2024-03-25T18:14:22.507 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.504+0000 7f3b72b77640 10 monclient: build_initial_monmap 2024-03-25T18:14:22.507 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.504+0000 7f3b72b77640 10 monclient: monmap: 2024-03-25T18:14:22.507 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:14:22.507 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:22.507 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:14:22.507 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:14:22.507 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:22.507 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:14:22.507 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:14:22.508 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:14:22.508 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:14:22.508 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:22.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.504+0000 7f3b72b77640 1 librados: starting msgr at 2024-03-25T18:14:22.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.504+0000 7f3b72b77640 1 librados: starting objecter 2024-03-25T18:14:22.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.504+0000 7f3b72b77640 1 Processor -- start 2024-03-25T18:14:22.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.504+0000 7f3b72b77640 1 -- start start 2024-03-25T18:14:22.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.504+0000 7f3b72b77640 1 librados: setting wanted keys 2024-03-25T18:14:22.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.504+0000 7f3b72b77640 1 librados: calling monclient init 2024-03-25T18:14:22.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.504+0000 7f3b72b77640 10 monclient: init 2024-03-25T18:14:22.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.505+0000 7f3b72b77640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:22.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.505+0000 7f3b72b77640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:22.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.505+0000 7f3b72b77640 10 monclient(hunting): picked mon.noname-b con 0x7f3b6c070570 addr v1:172.21.15.194:6789/0 2024-03-25T18:14:22.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.505+0000 7f3b72b77640 10 monclient(hunting): picked mon.noname-a con 0x7f3b6c0727a0 addr v1:172.21.15.156:6789/0 2024-03-25T18:14:22.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.505+0000 7f3b72b77640 10 monclient(hunting): picked mon.noname-c con 0x7f3b6c13fc60 addr v1:172.21.15.156:6790/0 2024-03-25T18:14:22.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.505+0000 7f3b72b77640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b6c1db200 con 0x7f3b6c0727a0 2024-03-25T18:14:22.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.505+0000 7f3b72b77640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b6c1414f0 con 0x7f3b6c070570 2024-03-25T18:14:22.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.505+0000 7f3b72b77640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b6c1dae00 con 0x7f3b6c13fc60 2024-03-25T18:14:22.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.505+0000 7f3b72b77640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:22.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.505+0000 7f3b72b77640 10 monclient(hunting): authenticate will time out at 7343.816547s 2024-03-25T18:14:22.518 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.517+0000 7f3b70913640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f3b6c070570 0x7f3b6c141c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:36018/0 (socket says 172.21.15.156:36018) 2024-03-25T18:14:22.518 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.517+0000 7f3b70913640 1 -- 172.21.15.156:0/2725466445 learned_addr learned my addr 172.21.15.156:0/2725466445 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:14:22.522 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.522+0000 7f3b697fa640 10 client.?.objecter ms_handle_connect 0x7f3b6c070570 2024-03-25T18:14:22.522 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.522+0000 7f3b697fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:22.530 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.529+0000 7f3b697fa640 10 client.?.objecter ms_handle_connect 0x7f3b6c13fc60 2024-03-25T18:14:22.530 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.529+0000 7f3b697fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:22.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.533+0000 7f3b697fa640 1 -- 172.21.15.156:0/2725466445 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 387969475 0 0) 0x7f3b6c1414f0 con 0x7f3b6c070570 2024-03-25T18:14:22.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.533+0000 7f3b697fa640 10 monclient(hunting): _negotiate 2024-03-25T18:14:22.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.533+0000 7f3b697fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:22.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.533+0000 7f3b697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:22.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.533+0000 7f3b697fa640 10 monclient(hunting): my global_id is 4501 2024-03-25T18:14:22.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.533+0000 7f3b697fa640 1 -- 172.21.15.156:0/2725466445 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b50004eb0 con 0x7f3b6c070570 2024-03-25T18:14:22.537 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.536+0000 7f3b697fa640 10 client.?.objecter ms_handle_connect 0x7f3b6c0727a0 2024-03-25T18:14:22.537 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.536+0000 7f3b697fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:22.539 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.538+0000 7f3b697fa640 1 -- 172.21.15.156:0/2725466445 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1924005019 0 0) 0x7f3b6c1dae00 con 0x7f3b6c13fc60 2024-03-25T18:14:22.539 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.538+0000 7f3b697fa640 10 monclient(hunting): _negotiate 2024-03-25T18:14:22.539 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.538+0000 7f3b697fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:22.539 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.538+0000 7f3b697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:22.539 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.538+0000 7f3b697fa640 10 monclient(hunting): my global_id is 4505 2024-03-25T18:14:22.540 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.538+0000 7f3b697fa640 1 -- 172.21.15.156:0/2725466445 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b6c1414f0 con 0x7f3b6c13fc60 2024-03-25T18:14:22.552 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.551+0000 7f3b697fa640 1 -- 172.21.15.156:0/2725466445 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 30382145 0 0) 0x7f3b6c1414f0 con 0x7f3b6c13fc60 2024-03-25T18:14:22.552 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.551+0000 7f3b697fa640 1 -- 172.21.15.156:0/2725466445 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b6c1dae00 con 0x7f3b6c13fc60 2024-03-25T18:14:22.552 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.551+0000 7f3b697fa640 1 -- 172.21.15.156:0/2725466445 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f3b64001e80 con 0x7f3b6c13fc60 2024-03-25T18:14:22.552 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.551+0000 7f3b697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:22.552 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.551+0000 7f3b697fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:22.552 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.551+0000 7f3b697fa640 10 monclient(hunting): dump: 2024-03-25T18:14:22.552 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:22.553 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:22.553 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:22.553 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:22.553 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:22.553 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:22.553 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:22.553 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:22.553 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:22.553 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:22.553 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.551+0000 7f3b697fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:22.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.553+0000 7f3b697fa640 1 -- 172.21.15.156:0/2725466445 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450036043 0 0) 0x7f3b6c1db200 con 0x7f3b6c0727a0 2024-03-25T18:14:22.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.553+0000 7f3b697fa640 10 monclient(hunting): _negotiate 2024-03-25T18:14:22.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.553+0000 7f3b697fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:22.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.553+0000 7f3b697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:22.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.553+0000 7f3b697fa640 10 monclient(hunting): my global_id is 4497 2024-03-25T18:14:22.554 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.553+0000 7f3b697fa640 1 -- 172.21.15.156:0/2725466445 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b6c1414f0 con 0x7f3b6c0727a0 2024-03-25T18:14:22.555 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.553+0000 7f3b6bfff640 1 -- 172.21.15.156:0/2725466445 >> v1:172.21.15.156:6789/0 conn(0x7f3b6c0727a0 legacy=0x7f3b6c1db690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T18:14:22.555 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.553+0000 7f3b6bfff640 1 -- 172.21.15.156:0/2725466445 >> v1:172.21.15.156:6789/0 conn(0x7f3b6c0727a0 legacy=0x7f3b6c1db690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:14:22.555 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.553+0000 7f3b6bfff640 1 --1- 172.21.15.156:0/2725466445 >> v1:172.21.15.156:6789/0 conn(0x7f3b6c0727a0 0x7f3b6c1db690 :-1 s=OPENED pgs=43237 cs=1 l=1).handle_message read tag failed 2024-03-25T18:14:22.555 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.553+0000 7f3b6bfff640 1 --1- 172.21.15.156:0/2725466445 >> v1:172.21.15.156:6789/0 conn(0x7f3b6c0727a0 0x7f3b6c1db690 :-1 s=OPENED pgs=43237 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:14:22.555 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.553+0000 7f3b697fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.156:6789/0 2024-03-25T18:14:22.555 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.554+0000 7f3b697fa640 1 -- 172.21.15.156:0/2725466445 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1370807286 0 0) 0x7f3b50004eb0 con 0x7f3b6c070570 2024-03-25T18:14:22.555 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.554+0000 7f3b697fa640 1 -- 172.21.15.156:0/2725466445 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b6c1db200 con 0x7f3b6c070570 2024-03-25T18:14:22.555 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.554+0000 7f3b697fa640 1 -- 172.21.15.156:0/2725466445 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f3b60001970 con 0x7f3b6c070570 2024-03-25T18:14:22.555 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.554+0000 7f3b697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:22.555 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.554+0000 7f3b697fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:22.556 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.554+0000 7f3b697fa640 10 monclient(hunting): dump: 2024-03-25T18:14:22.556 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:22.556 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:22.556 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:22.556 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:22.556 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:22.556 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:22.556 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:22.556 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:22.556 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:22.556 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:22.563 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.562+0000 7f3b697fa640 1 -- 172.21.15.156:0/2725466445 <== mon.2 v1:172.21.15.156:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1419347021 0 0) 0x7f3b6c1dae00 con 0x7f3b6c13fc60 2024-03-25T18:14:22.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.562+0000 7f3b697fa640 1 -- 172.21.15.156:0/2725466445 >> v1:172.21.15.194:6789/0 conn(0x7f3b6c070570 legacy=0x7f3b6c141c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:22.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.562+0000 7f3b697fa640 1 -- 172.21.15.156:0/2725466445 >> v1:172.21.15.156:6789/0 conn(0x7f3b6c0727a0 legacy=0x7f3b6c1db690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:14:22.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.562+0000 7f3b697fa640 10 monclient: _finish_hunting 0 2024-03-25T18:14:22.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.562+0000 7f3b697fa640 1 monclient: found mon.c 2024-03-25T18:14:22.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.562+0000 7f3b697fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:22.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.562+0000 7f3b697fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:14:22.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.562+0000 7f3b697fa640 1 -- 172.21.15.156:0/2725466445 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b6c1322a0 con 0x7f3b6c13fc60 2024-03-25T18:14:22.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.562+0000 7f3b697fa640 10 monclient: _finish_auth 0 2024-03-25T18:14:22.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.562+0000 7f3b697fa640 10 monclient: _check_auth_tickets 2024-03-25T18:14:22.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.562+0000 7f3b697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:22.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.562+0000 7f3b72b77640 5 monclient: authenticate success, global_id 4505 2024-03-25T18:14:22.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.562+0000 7f3b72b77640 10 monclient: _renew_subs 2024-03-25T18:14:22.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.562+0000 7f3b72b77640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:14:22.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.562+0000 7f3b72b77640 1 -- 172.21.15.156:0/2725466445 --> v1:172.21.15.156:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b6c1324f0 con 0x7f3b6c13fc60 2024-03-25T18:14:22.564 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.562+0000 7f3b72b77640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:22.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.562+0000 7f3b72b77640 10 monclient: _renew_subs 2024-03-25T18:14:22.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.562+0000 7f3b72b77640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:14:22.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.562+0000 7f3b72b77640 1 -- 172.21.15.156:0/2725466445 --> v1:172.21.15.156:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b6c1410e0 con 0x7f3b6c13fc60 2024-03-25T18:14:22.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.562+0000 7f3b72b77640 1 librados: init done 2024-03-25T18:14:22.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.563+0000 7f3b72b77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:22.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.563+0000 7f3b72b77640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:22.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.563+0000 7f3b72b77640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:14:22.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.563+0000 7f3b72b77640 1 -- 172.21.15.156:0/2725466445 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b6c0627b0 con 0x7f3b6c13fc60 2024-03-25T18:14:22.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.862+0000 7f3b697fa640 1 -- 172.21.15.156:0/2725466445 <== mon.2 v1:172.21.15.156:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b64001460 con 0x7f3b6c13fc60 2024-03-25T18:14:22.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.862+0000 7f3b697fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:22.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.862+0000 7f3b697fa640 1 -- 172.21.15.156:0/2725466445 <== mon.2 v1:172.21.15.156:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f3b64003dc0 con 0x7f3b6c13fc60 2024-03-25T18:14:22.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.862+0000 7f3b697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:22.864 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.862+0000 7f3b697fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:22.864 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.862+0000 7f3b697fa640 10 monclient: dump: 2024-03-25T18:14:22.864 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:22.864 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:22.864 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:22.864 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:22.864 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:22.864 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:22.864 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:22.864 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:22.864 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:22.864 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:22.864 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.862+0000 7f3b6affd640 4 set_mon_vals no callback set 2024-03-25T18:14:22.865 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.863+0000 7f3b697fa640 1 -- 172.21.15.156:0/2725466445 <== mon.2 v1:172.21.15.156:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f3b64019d90 con 0x7f3b6c13fc60 2024-03-25T18:14:22.865 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.863+0000 7f3b70913640 10 monclient: get_auth_request con 0x7f3b5006de80 auth_method 0 2024-03-25T18:14:22.865 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.864+0000 7f3b697fa640 10 client.4505.objecter ms_handle_connect 0x7f3b5006de80 2024-03-25T18:14:22.870 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.868+0000 7f3b697fa640 1 -- 172.21.15.156:0/2725466445 <== mon.2 v1:172.21.15.156:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f3b640538e0 con 0x7f3b6c13fc60 2024-03-25T18:14:22.870 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.868+0000 7f3b697fa640 10 client.4505.objecter ms_dispatch 0x7f3b6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:22.870 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.868+0000 7f3b697fa640 3 client.4505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:22.870 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.868+0000 7f3b697fa640 3 client.4505.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:22.870 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.869+0000 7f3b697fa640 20 client.4505.objecter dump_active .. 0 homeless 2024-03-25T18:14:22.870 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.869+0000 7f3b697fa640 1 -- 172.21.15.156:0/2725466445 <== mon.2 v1:172.21.15.156:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3b6404f3f0 con 0x7f3b6c13fc60 2024-03-25T18:14:22.870 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.869+0000 7f3b697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:22.870 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:22.869+0000 7f3b697fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:23.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:23.119+0000 7f3b72b77640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:23.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:23.119+0000 7f3b72b77640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:23.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:23.119+0000 7f3b72b77640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:14:23.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:23.119+0000 7f3b72b77640 1 -- 172.21.15.156:0/2725466445 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b6c0b5380 con 0x7f3b6c13fc60 2024-03-25T18:14:23.146 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:23.145+0000 7f3b697fa640 1 -- 172.21.15.156:0/2725466445 <== mon.2 v1:172.21.15.156:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f3b64051320 con 0x7f3b6c13fc60 2024-03-25T18:14:23.146 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:23.145+0000 7f3b697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:23.146 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:23.145+0000 7f3b697fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:23.147 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:14:23.147 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:23.149 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:23.147+0000 7f3b72b77640 10 librados: watch_flush enter 2024-03-25T18:14:23.149 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:23.147+0000 7f3b72b77640 10 librados: watch_flush exit 2024-03-25T18:14:23.149 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:23.147+0000 7f3b72b77640 20 client.4505.objecter shutdown clearing up homeless session... 2024-03-25T18:14:23.149 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:23.147+0000 7f3b72b77640 10 client.4505.objecter successfully canceled tick 2024-03-25T18:14:23.149 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:23.147+0000 7f3b72b77640 1 -- 172.21.15.156:0/2725466445 >> v1:172.21.15.194:6800/2946427713 conn(0x7f3b5006de80 legacy=0x7f3b50070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:23.149 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:23.147+0000 7f3b72b77640 10 monclient: shutdown 2024-03-25T18:14:23.149 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:23.147+0000 7f3b72b77640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:23.150 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:23.147+0000 7f3b72b77640 1 -- 172.21.15.156:0/2725466445 >> v1:172.21.15.156:6790/0 conn(0x7f3b6c13fc60 legacy=0x7f3b6c131b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:23.150 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:23.148+0000 7f3b72b77640 1 -- 172.21.15.156:0/2725466445 shutdown_connections 2024-03-25T18:14:23.150 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:23.148+0000 7f3b72b77640 1 -- 172.21.15.156:0/2725466445 >> 172.21.15.156:0/2725466445 conn(0x7f3b6c0b4a80 msgr2=0x7f3b6c06b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:23.150 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:23.148+0000 7f3b72b77640 1 -- 172.21.15.156:0/2725466445 shutdown_connections 2024-03-25T18:14:23.150 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:23.148+0000 7f3b72b77640 1 -- 172.21.15.156:0/2725466445 wait complete. 2024-03-25T18:14:23.150 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:23.148+0000 7f3b72b77640 1 librados: shutdown 2024-03-25T18:14:23.162 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:24.268 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:06:30.258 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:24.286 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:06:30.276 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:24.308 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:06:30.298 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:24.336 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:06:30.327 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:24.338 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:06:30.305 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:24.349 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:06:30.316 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:24.350 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:06:30.340 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:24.382 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:06:30.349 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:24.391 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:06:30.381 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:29.968 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:30.082 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.081+0000 7fa15a5db640 10 monclient: get_monmap_and_config 2024-03-25T18:14:30.083 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.081+0000 7fa15a5db640 10 monclient: build_initial_monmap 2024-03-25T18:14:30.083 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.081+0000 7fa15a5db640 10 monclient: monmap: 2024-03-25T18:14:30.083 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:14:30.083 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:30.083 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:14:30.082712+0000 2024-03-25T18:14:30.083 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:14:30.082712+0000 2024-03-25T18:14:30.083 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:30.083 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:30.083 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:14:30.083 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:14:30.083 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:14:30.083 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:30.084 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.082+0000 7fa15a5db640 1 Processor -- start 2024-03-25T18:14:30.084 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.082+0000 7fa15a5db640 1 -- start start 2024-03-25T18:14:30.084 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.082+0000 7fa15a5db640 10 monclient: init 2024-03-25T18:14:30.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.083+0000 7fa15a5db640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:30.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.083+0000 7fa15a5db640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:30.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.083+0000 7fa15a5db640 10 monclient(hunting): picked mon.noname-b con 0x7fa1540b4d60 addr v1:172.21.15.156:6790/0 2024-03-25T18:14:30.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.083+0000 7fa15a5db640 10 monclient(hunting): picked mon.noname-a con 0x7fa1540b1730 addr v1:172.21.15.156:6789/0 2024-03-25T18:14:30.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.083+0000 7fa15a5db640 10 monclient(hunting): picked mon.noname-c con 0x7fa15414be90 addr v1:172.21.15.194:6789/0 2024-03-25T18:14:30.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.083+0000 7fa15a5db640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1540af940 con 0x7fa1540b1730 2024-03-25T18:14:30.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.083+0000 7fa15a5db640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1540b0a70 con 0x7fa15414be90 2024-03-25T18:14:30.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.083+0000 7fa15a5db640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1540b0d50 con 0x7fa1540b4d60 2024-03-25T18:14:30.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.083+0000 7fa15a5db640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:30.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.083+0000 7fa15a5db640 10 monclient(hunting): authenticate will time out at 7351.394283s 2024-03-25T18:14:30.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.092+0000 7fa158b78640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fa15414be90 0x7fa15414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:40726/0 (socket says 172.21.15.156:40726) 2024-03-25T18:14:30.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.092+0000 7fa158b78640 1 -- 172.21.15.156:0/4234574846 learned_addr learned my addr 172.21.15.156:0/4234574846 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:14:30.116 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.115+0000 7fa152ffd640 1 -- 172.21.15.156:0/4234574846 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349453665 0 0) 0x7fa1540b0d50 con 0x7fa1540b4d60 2024-03-25T18:14:30.116 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.115+0000 7fa152ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:30.116 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.115+0000 7fa152ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:30.116 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.115+0000 7fa152ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:30.117 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.115+0000 7fa152ffd640 10 monclient(hunting): my global_id is 4508 2024-03-25T18:14:30.117 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.115+0000 7fa152ffd640 1 -- 172.21.15.156:0/4234574846 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa13c003520 con 0x7fa1540b4d60 2024-03-25T18:14:30.124 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.123+0000 7fa152ffd640 1 -- 172.21.15.156:0/4234574846 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318377038 0 0) 0x7fa1540b0a70 con 0x7fa15414be90 2024-03-25T18:14:30.124 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.123+0000 7fa152ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:30.124 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.123+0000 7fa152ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:30.124 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.123+0000 7fa152ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:30.124 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.123+0000 7fa152ffd640 10 monclient(hunting): my global_id is 4504 2024-03-25T18:14:30.124 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.123+0000 7fa152ffd640 1 -- 172.21.15.156:0/4234574846 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1540b0d50 con 0x7fa15414be90 2024-03-25T18:14:30.130 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.129+0000 7fa152ffd640 1 -- 172.21.15.156:0/4234574846 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913353687 0 0) 0x7fa1540af940 con 0x7fa1540b1730 2024-03-25T18:14:30.130 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.129+0000 7fa152ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:30.130 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.129+0000 7fa152ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:30.130 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.129+0000 7fa152ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:30.130 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.129+0000 7fa152ffd640 10 monclient(hunting): my global_id is 4500 2024-03-25T18:14:30.130 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.129+0000 7fa152ffd640 1 -- 172.21.15.156:0/4234574846 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1540b0a70 con 0x7fa1540b1730 2024-03-25T18:14:30.142 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.141+0000 7fa152ffd640 1 -- 172.21.15.156:0/4234574846 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3366048661 0 0) 0x7fa13c003520 con 0x7fa1540b4d60 2024-03-25T18:14:30.142 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.141+0000 7fa152ffd640 1 -- 172.21.15.156:0/4234574846 >> v1:172.21.15.194:6789/0 conn(0x7fa15414be90 legacy=0x7fa15414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:30.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.141+0000 7fa152ffd640 1 -- 172.21.15.156:0/4234574846 >> v1:172.21.15.156:6789/0 conn(0x7fa1540b1730 legacy=0x7fa1540b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:30.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.141+0000 7fa152ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:30.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.141+0000 7fa152ffd640 1 monclient: found mon.noname-b 2024-03-25T18:14:30.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.141+0000 7fa152ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:30.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.141+0000 7fa152ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.156:6790/0 2024-03-25T18:14:30.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.141+0000 7fa152ffd640 1 -- 172.21.15.156:0/4234574846 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1540b1030 con 0x7fa1540b4d60 2024-03-25T18:14:30.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.141+0000 7fa152ffd640 10 monclient: _finish_auth 0 2024-03-25T18:14:30.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.141+0000 7fa152ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:30.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.141+0000 7fa152ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:30.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.141+0000 7fa152ffd640 1 -- 172.21.15.156:0/4234574846 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fa144001690 con 0x7fa1540b4d60 2024-03-25T18:14:30.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.141+0000 7fa152ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:30.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.141+0000 7fa152ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:30.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.141+0000 7fa152ffd640 10 monclient: dump: 2024-03-25T18:14:30.143 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:30.144 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:30.144 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:30.144 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:30.144 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:30.144 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:30.144 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:30.144 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:30.144 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:30.144 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:30.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.141+0000 7fa15a5db640 5 monclient: authenticate success, global_id 4508 2024-03-25T18:14:30.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.141+0000 7fa15a5db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:30.166 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.165+0000 7fa152ffd640 1 -- 172.21.15.156:0/4234574846 <== mon.2 v1:172.21.15.156:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1440017f0 con 0x7fa1540b4d60 2024-03-25T18:14:30.166 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.165+0000 7fa152ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:30.166 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.165+0000 7fa152ffd640 1 -- 172.21.15.156:0/4234574846 <== mon.2 v1:172.21.15.156:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fa144001c00 con 0x7fa1540b4d60 2024-03-25T18:14:30.166 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.165+0000 7fa15a5db640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:30.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.165+0000 7fa15a5db640 4 set_mon_vals no callback set 2024-03-25T18:14:30.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.165+0000 7fa152ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:30.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.165+0000 7fa152ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:30.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.165+0000 7fa152ffd640 10 monclient: dump: 2024-03-25T18:14:30.167 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:30.167 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:30.167 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:30.167 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:30.167 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:30.167 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:30.167 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:30.167 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:30.167 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:30.168 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:30.170 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.166+0000 7fa15a5db640 10 monclient: shutdown 2024-03-25T18:14:30.170 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.166+0000 7fa15a5db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:30.170 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.166+0000 7fa15a5db640 1 -- 172.21.15.156:0/4234574846 >> v1:172.21.15.156:6790/0 conn(0x7fa1540b4d60 legacy=0x7fa15413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:30.170 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.166+0000 7fa15a5db640 1 -- 172.21.15.156:0/4234574846 shutdown_connections 2024-03-25T18:14:30.170 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.166+0000 7fa15a5db640 1 -- 172.21.15.156:0/4234574846 >> 172.21.15.156:0/4234574846 conn(0x7fa1540617a0 msgr2=0x7fa154061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:30.170 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.166+0000 7fa15a5db640 1 -- 172.21.15.156:0/4234574846 shutdown_connections 2024-03-25T18:14:30.170 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.166+0000 7fa15a5db640 1 -- 172.21.15.156:0/4234574846 wait complete. 2024-03-25T18:14:30.170 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.167+0000 7fa15a5db640 10 monclient: build_initial_monmap 2024-03-25T18:14:30.170 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.167+0000 7fa15a5db640 10 monclient: monmap: 2024-03-25T18:14:30.170 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:14:30.170 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:30.170 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:14:30.170 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:14:30.171 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:30.171 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:14:30.171 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:14:30.171 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:14:30.171 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:14:30.171 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:30.171 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.167+0000 7fa15a5db640 1 librados: starting msgr at 2024-03-25T18:14:30.171 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.167+0000 7fa15a5db640 1 librados: starting objecter 2024-03-25T18:14:30.171 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.167+0000 7fa15a5db640 1 Processor -- start 2024-03-25T18:14:30.171 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.167+0000 7fa15a5db640 1 -- start start 2024-03-25T18:14:30.171 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.167+0000 7fa15a5db640 1 librados: setting wanted keys 2024-03-25T18:14:30.171 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.167+0000 7fa15a5db640 1 librados: calling monclient init 2024-03-25T18:14:30.171 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.167+0000 7fa15a5db640 10 monclient: init 2024-03-25T18:14:30.171 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.168+0000 7fa15a5db640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:30.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.168+0000 7fa15a5db640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:30.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.168+0000 7fa15a5db640 10 monclient(hunting): picked mon.noname-a con 0x7fa1540b1730 addr v1:172.21.15.156:6789/0 2024-03-25T18:14:30.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.168+0000 7fa15a5db640 10 monclient(hunting): picked mon.noname-c con 0x7fa1540b4d60 addr v1:172.21.15.156:6790/0 2024-03-25T18:14:30.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.168+0000 7fa15a5db640 10 monclient(hunting): picked mon.noname-b con 0x7fa15414be90 addr v1:172.21.15.194:6789/0 2024-03-25T18:14:30.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.168+0000 7fa15a5db640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa154136870 con 0x7fa1540b1730 2024-03-25T18:14:30.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.168+0000 7fa15a5db640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa154136350 con 0x7fa15414be90 2024-03-25T18:14:30.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.168+0000 7fa15a5db640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa154136fa0 con 0x7fa1540b4d60 2024-03-25T18:14:30.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.168+0000 7fa15a5db640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:30.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.168+0000 7fa15a5db640 10 monclient(hunting): authenticate will time out at 7351.479472s 2024-03-25T18:14:30.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.177+0000 7fa153fff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fa1540b1730 0x7fa15412fcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:39544/0 (socket says 172.21.15.156:39544) 2024-03-25T18:14:30.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.178+0000 7fa153fff640 1 -- 172.21.15.156:0/3444557853 learned_addr learned my addr 172.21.15.156:0/3444557853 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:14:30.180 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.179+0000 7fa150ff9640 10 client.?.objecter ms_handle_connect 0x7fa1540b1730 2024-03-25T18:14:30.180 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.179+0000 7fa150ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:30.193 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.192+0000 7fa150ff9640 10 client.?.objecter ms_handle_connect 0x7fa1540b4d60 2024-03-25T18:14:30.193 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.192+0000 7fa150ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:30.199 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.198+0000 7fa150ff9640 1 -- 172.21.15.156:0/3444557853 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153559444 0 0) 0x7fa154136870 con 0x7fa1540b1730 2024-03-25T18:14:30.199 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.198+0000 7fa150ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:30.199 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.198+0000 7fa150ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:30.199 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.198+0000 7fa150ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:30.199 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.198+0000 7fa150ff9640 10 monclient(hunting): my global_id is 4503 2024-03-25T18:14:30.199 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.198+0000 7fa150ff9640 1 -- 172.21.15.156:0/3444557853 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa140004eb0 con 0x7fa1540b1730 2024-03-25T18:14:30.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.199+0000 7fa150ff9640 10 client.?.objecter ms_handle_connect 0x7fa15414be90 2024-03-25T18:14:30.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.199+0000 7fa150ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:30.208 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.207+0000 7fa150ff9640 1 -- 172.21.15.156:0/3444557853 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2868585843 0 0) 0x7fa154136350 con 0x7fa15414be90 2024-03-25T18:14:30.208 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.207+0000 7fa150ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:30.208 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.207+0000 7fa150ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:30.208 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.207+0000 7fa150ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:30.208 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.207+0000 7fa150ff9640 10 monclient(hunting): my global_id is 4507 2024-03-25T18:14:30.208 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.207+0000 7fa150ff9640 1 -- 172.21.15.156:0/3444557853 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa154136870 con 0x7fa15414be90 2024-03-25T18:14:30.221 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.220+0000 7fa150ff9640 1 -- 172.21.15.156:0/3444557853 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3474056959 0 0) 0x7fa154136870 con 0x7fa15414be90 2024-03-25T18:14:30.222 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.220+0000 7fa150ff9640 1 -- 172.21.15.156:0/3444557853 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa154136350 con 0x7fa15414be90 2024-03-25T18:14:30.222 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.220+0000 7fa150ff9640 1 -- 172.21.15.156:0/3444557853 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fa14c001b50 con 0x7fa15414be90 2024-03-25T18:14:30.222 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.220+0000 7fa150ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:30.222 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.221+0000 7fa150ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:30.222 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.221+0000 7fa150ff9640 10 monclient(hunting): dump: 2024-03-25T18:14:30.222 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:30.222 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:30.222 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:30.222 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:30.222 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:30.222 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:30.222 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:30.222 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:30.223 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:30.223 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:30.223 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.221+0000 7fa150ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:30.227 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.226+0000 7fa150ff9640 1 -- 172.21.15.156:0/3444557853 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009829986 0 0) 0x7fa154136fa0 con 0x7fa1540b4d60 2024-03-25T18:14:30.227 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.226+0000 7fa150ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:30.227 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.226+0000 7fa150ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:30.227 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.226+0000 7fa150ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:30.227 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.226+0000 7fa150ff9640 10 monclient(hunting): my global_id is 4511 2024-03-25T18:14:30.227 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.226+0000 7fa150ff9640 1 -- 172.21.15.156:0/3444557853 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa154136870 con 0x7fa1540b4d60 2024-03-25T18:14:30.228 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.226+0000 7fa150ff9640 1 -- 172.21.15.156:0/3444557853 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1211818481 0 0) 0x7fa140004eb0 con 0x7fa1540b1730 2024-03-25T18:14:30.228 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.226+0000 7fa150ff9640 1 -- 172.21.15.156:0/3444557853 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa154136fa0 con 0x7fa1540b1730 2024-03-25T18:14:30.228 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.226+0000 7fa150ff9640 1 -- 172.21.15.156:0/3444557853 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fa144001bc0 con 0x7fa1540b1730 2024-03-25T18:14:30.228 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.226+0000 7fa150ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:30.228 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.226+0000 7fa150ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:30.228 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.226+0000 7fa150ff9640 10 monclient(hunting): dump: 2024-03-25T18:14:30.228 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:30.228 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:30.228 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:30.228 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:30.228 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:30.229 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:30.229 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:30.229 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:30.229 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:30.229 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:30.324 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.322+0000 7fa150ff9640 1 -- 172.21.15.156:0/3444557853 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1842578108 0 0) 0x7fa154136fa0 con 0x7fa1540b1730 2024-03-25T18:14:30.324 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.322+0000 7fa150ff9640 1 -- 172.21.15.156:0/3444557853 >> v1:172.21.15.156:6790/0 conn(0x7fa1540b4d60 legacy=0x7fa1541dac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:30.324 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.322+0000 7fa150ff9640 1 -- 172.21.15.156:0/3444557853 >> v1:172.21.15.194:6789/0 conn(0x7fa15414be90 legacy=0x7fa1541e2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:30.324 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.322+0000 7fa150ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:14:30.324 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.322+0000 7fa150ff9640 1 monclient: found mon.a 2024-03-25T18:14:30.324 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.322+0000 7fa150ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:30.324 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.322+0000 7fa150ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:14:30.324 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.322+0000 7fa150ff9640 1 -- 172.21.15.156:0/3444557853 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa154138620 con 0x7fa1540b1730 2024-03-25T18:14:30.324 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.322+0000 7fa150ff9640 10 monclient: _finish_auth 0 2024-03-25T18:14:30.324 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.322+0000 7fa150ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:14:30.324 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.322+0000 7fa150ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:30.324 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.322+0000 7fa15a5db640 5 monclient: authenticate success, global_id 4503 2024-03-25T18:14:30.325 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.322+0000 7fa15a5db640 10 monclient: _renew_subs 2024-03-25T18:14:30.325 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.322+0000 7fa15a5db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:14:30.325 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.322+0000 7fa15a5db640 1 -- 172.21.15.156:0/3444557853 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1541324e0 con 0x7fa1540b1730 2024-03-25T18:14:30.325 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.322+0000 7fa15a5db640 10 client.4503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:30.325 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.322+0000 7fa15a5db640 10 monclient: _renew_subs 2024-03-25T18:14:30.325 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.322+0000 7fa15a5db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:14:30.325 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.322+0000 7fa15a5db640 1 -- 172.21.15.156:0/3444557853 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa154135d30 con 0x7fa1540b1730 2024-03-25T18:14:30.325 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.322+0000 7fa15a5db640 1 librados: init done 2024-03-25T18:14:30.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.323+0000 7fa15a5db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:30.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.323+0000 7fa15a5db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:30.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.323+0000 7fa15a5db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:14:30.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.323+0000 7fa15a5db640 1 -- 172.21.15.156:0/3444557853 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1540627b0 con 0x7fa1540b1730 2024-03-25T18:14:30.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.588+0000 7fa150ff9640 1 -- 172.21.15.156:0/3444557853 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1440011f0 con 0x7fa1540b1730 2024-03-25T18:14:30.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.588+0000 7fa150ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:30.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.588+0000 7fa150ff9640 1 -- 172.21.15.156:0/3444557853 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fa144003b00 con 0x7fa1540b1730 2024-03-25T18:14:30.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.588+0000 7fa150ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:30.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.588+0000 7fa150ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:30.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.588+0000 7fa150ff9640 10 monclient: dump: 2024-03-25T18:14:30.590 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:30.590 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:30.590 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:30.590 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:30.590 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:30.590 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:30.590 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:30.590 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:30.591 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:30.591 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:30.591 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.588+0000 7fa1527fc640 4 set_mon_vals no callback set 2024-03-25T18:14:30.591 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.589+0000 7fa150ff9640 1 -- 172.21.15.156:0/3444557853 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7fa144003d20 con 0x7fa1540b1730 2024-03-25T18:14:30.591 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.590+0000 7fa1537fe640 10 monclient: get_auth_request con 0x7fa14006df70 auth_method 0 2024-03-25T18:14:30.591 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.590+0000 7fa150ff9640 10 client.4503.objecter ms_handle_connect 0x7fa14006df70 2024-03-25T18:14:30.601 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.600+0000 7fa150ff9640 1 -- 172.21.15.156:0/3444557853 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7fa1440548d0 con 0x7fa1540b1730 2024-03-25T18:14:30.601 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.600+0000 7fa150ff9640 10 client.4503.objecter ms_dispatch 0x7fa154000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:30.601 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.600+0000 7fa150ff9640 3 client.4503.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:30.601 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.600+0000 7fa150ff9640 3 client.4503.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:30.601 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.600+0000 7fa150ff9640 20 client.4503.objecter dump_active .. 0 homeless 2024-03-25T18:14:30.601 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.600+0000 7fa150ff9640 1 -- 172.21.15.156:0/3444557853 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa1440503e0 con 0x7fa1540b1730 2024-03-25T18:14:30.601 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.600+0000 7fa150ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:30.601 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.600+0000 7fa150ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:30.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.831+0000 7fa15a5db640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:30.833 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.831+0000 7fa15a5db640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:30.833 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.831+0000 7fa15a5db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:14:30.833 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.831+0000 7fa15a5db640 1 -- 172.21.15.156:0/3444557853 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1541387b0 con 0x7fa1540b1730 2024-03-25T18:14:30.878 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.877+0000 7fa150ff9640 1 -- 172.21.15.156:0/3444557853 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7fa144052310 con 0x7fa1540b1730 2024-03-25T18:14:30.878 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.877+0000 7fa150ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:30.879 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.877+0000 7fa150ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:30.879 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:14:30.879 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:30.882 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.880+0000 7fa15a5db640 10 librados: watch_flush enter 2024-03-25T18:14:30.882 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.880+0000 7fa15a5db640 10 librados: watch_flush exit 2024-03-25T18:14:30.882 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.880+0000 7fa15a5db640 20 client.4503.objecter shutdown clearing up homeless session... 2024-03-25T18:14:30.882 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.880+0000 7fa15a5db640 10 client.4503.objecter successfully canceled tick 2024-03-25T18:14:30.882 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.881+0000 7fa15a5db640 1 -- 172.21.15.156:0/3444557853 >> v1:172.21.15.194:6800/2946427713 conn(0x7fa14006df70 legacy=0x7fa140070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:30.883 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.881+0000 7fa15a5db640 10 monclient: shutdown 2024-03-25T18:14:30.883 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.881+0000 7fa15a5db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:30.883 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.881+0000 7fa15a5db640 1 -- 172.21.15.156:0/3444557853 >> v1:172.21.15.156:6789/0 conn(0x7fa1540b1730 legacy=0x7fa15412fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:30.883 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.881+0000 7fa15a5db640 1 -- 172.21.15.156:0/3444557853 shutdown_connections 2024-03-25T18:14:30.883 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.881+0000 7fa15a5db640 1 -- 172.21.15.156:0/3444557853 >> 172.21.15.156:0/3444557853 conn(0x7fa1540617a0 msgr2=0x7fa1540b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:30.883 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.881+0000 7fa15a5db640 1 -- 172.21.15.156:0/3444557853 shutdown_connections 2024-03-25T18:14:30.883 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.881+0000 7fa15a5db640 1 -- 172.21.15.156:0/3444557853 wait complete. 2024-03-25T18:14:30.883 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:30.881+0000 7fa15a5db640 1 librados: shutdown 2024-03-25T18:14:30.894 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:37.702 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:37.817 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.816+0000 7f63b718c640 10 monclient: get_monmap_and_config 2024-03-25T18:14:37.817 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.816+0000 7f63b718c640 10 monclient: build_initial_monmap 2024-03-25T18:14:37.818 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.816+0000 7f63b718c640 10 monclient: monmap: 2024-03-25T18:14:37.818 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:14:37.818 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:37.818 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:14:37.817435+0000 2024-03-25T18:14:37.818 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:14:37.817435+0000 2024-03-25T18:14:37.818 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:37.818 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:37.818 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:14:37.818 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:14:37.818 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:14:37.819 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:37.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.817+0000 7f63b718c640 1 Processor -- start 2024-03-25T18:14:37.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.817+0000 7f63b718c640 1 -- start start 2024-03-25T18:14:37.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.817+0000 7f63b718c640 10 monclient: init 2024-03-25T18:14:37.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.818+0000 7f63b718c640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:37.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.818+0000 7f63b718c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:37.821 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.818+0000 7f63b718c640 10 monclient(hunting): picked mon.noname-b con 0x7f63b00a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:14:37.821 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.818+0000 7f63b718c640 10 monclient(hunting): picked mon.noname-c con 0x7f63b013f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:14:37.821 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.818+0000 7f63b718c640 10 monclient(hunting): picked mon.noname-a con 0x7f63b01430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:14:37.821 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.818+0000 7f63b718c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b0142520 con 0x7f63b01430c0 2024-03-25T18:14:37.821 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.819+0000 7f63b718c640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b00b28a0 con 0x7f63b013f710 2024-03-25T18:14:37.821 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.819+0000 7f63b718c640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b00b2a30 con 0x7f63b00a98f0 2024-03-25T18:14:37.821 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.819+0000 7f63b718c640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:37.821 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.819+0000 7f63b718c640 10 monclient(hunting): authenticate will time out at 7359.129842s 2024-03-25T18:14:37.830 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.829+0000 7f63affff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f63b013f710 0x7f63b01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:46088/0 (socket says 172.21.15.156:46088) 2024-03-25T18:14:37.830 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.829+0000 7f63affff640 1 -- 172.21.15.156:0/2193020511 learned_addr learned my addr 172.21.15.156:0/2193020511 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:14:37.838 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.836+0000 7f63af7fe640 1 -- 172.21.15.156:0/2193020511 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2256446017 0 0) 0x7f63b00b2a30 con 0x7f63b00a98f0 2024-03-25T18:14:37.838 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.837+0000 7f63af7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:37.838 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.837+0000 7f63af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:37.838 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.837+0000 7f63af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:37.838 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.837+0000 7f63af7fe640 10 monclient(hunting): my global_id is 4514 2024-03-25T18:14:37.838 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.837+0000 7f63af7fe640 1 -- 172.21.15.156:0/2193020511 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63a0003520 con 0x7f63b00a98f0 2024-03-25T18:14:37.845 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.843+0000 7f63af7fe640 1 -- 172.21.15.156:0/2193020511 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1073705085 0 0) 0x7f63a0003520 con 0x7f63b00a98f0 2024-03-25T18:14:37.845 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.843+0000 7f63af7fe640 1 -- 172.21.15.156:0/2193020511 >> v1:172.21.15.194:6789/0 conn(0x7f63b013f710 legacy=0x7f63b01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:37.845 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.843+0000 7f63af7fe640 1 -- 172.21.15.156:0/2193020511 >> v1:172.21.15.156:6789/0 conn(0x7f63b01430c0 legacy=0x7f63b014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:14:37.845 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.844+0000 7f63af7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:14:37.845 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.844+0000 7f63af7fe640 1 monclient: found mon.noname-b 2024-03-25T18:14:37.845 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.844+0000 7f63af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:37.845 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.844+0000 7f63af7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.156:6790/0 2024-03-25T18:14:37.845 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.844+0000 7f63af7fe640 1 -- 172.21.15.156:0/2193020511 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b00b44c0 con 0x7f63b00a98f0 2024-03-25T18:14:37.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.844+0000 7f63af7fe640 10 monclient: _finish_auth 0 2024-03-25T18:14:37.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.844+0000 7f63af7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:14:37.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.844+0000 7f63af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:37.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.844+0000 7f63b718c640 5 monclient: authenticate success, global_id 4514 2024-03-25T18:14:37.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.844+0000 7f63b718c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:37.849 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.848+0000 7f63af7fe640 1 -- 172.21.15.156:0/2193020511 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f63a4001690 con 0x7f63b00a98f0 2024-03-25T18:14:37.849 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.848+0000 7f63af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:37.849 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.848+0000 7f63af7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:37.849 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.848+0000 7f63af7fe640 10 monclient: dump: 2024-03-25T18:14:37.849 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:37.850 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:37.850 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:37.850 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:37.850 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:37.850 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:37.850 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:37.850 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:37.850 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:37.850 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:37.850 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.848+0000 7f63b718c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:37.859 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.858+0000 7f63af7fe640 1 -- 172.21.15.156:0/2193020511 <== mon.2 v1:172.21.15.156:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63a4001960 con 0x7f63b00a98f0 2024-03-25T18:14:37.860 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.858+0000 7f63af7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:37.860 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.858+0000 7f63af7fe640 1 -- 172.21.15.156:0/2193020511 <== mon.2 v1:172.21.15.156:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f63a4001cf0 con 0x7f63b00a98f0 2024-03-25T18:14:37.860 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.858+0000 7f63b718c640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:37.860 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.858+0000 7f63b718c640 4 set_mon_vals no callback set 2024-03-25T18:14:37.860 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.858+0000 7f63af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:37.860 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.858+0000 7f63af7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:37.860 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.858+0000 7f63af7fe640 10 monclient: dump: 2024-03-25T18:14:37.860 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:37.860 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:37.861 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:37.861 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:37.861 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:37.861 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:37.861 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:37.861 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:37.861 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:37.861 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:37.864 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.859+0000 7f63b718c640 10 monclient: shutdown 2024-03-25T18:14:37.864 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.859+0000 7f63b718c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:37.864 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.859+0000 7f63b718c640 1 -- 172.21.15.156:0/2193020511 >> v1:172.21.15.156:6790/0 conn(0x7f63b00a98f0 legacy=0x7f63b00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:37.864 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.859+0000 7f63b718c640 1 -- 172.21.15.156:0/2193020511 shutdown_connections 2024-03-25T18:14:37.864 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.859+0000 7f63b718c640 1 -- 172.21.15.156:0/2193020511 >> 172.21.15.156:0/2193020511 conn(0x7f63b00b7ea0 msgr2=0x7f63b00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:37.864 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.859+0000 7f63b718c640 1 -- 172.21.15.156:0/2193020511 shutdown_connections 2024-03-25T18:14:37.864 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.859+0000 7f63b718c640 1 -- 172.21.15.156:0/2193020511 wait complete. 2024-03-25T18:14:37.864 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.860+0000 7f63b718c640 10 monclient: build_initial_monmap 2024-03-25T18:14:37.865 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.860+0000 7f63b718c640 10 monclient: monmap: 2024-03-25T18:14:37.865 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:14:37.865 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:37.865 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:14:37.865 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:14:37.865 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:37.865 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:14:37.865 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:14:37.865 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:14:37.865 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:14:37.865 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:37.866 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.861+0000 7f63b718c640 1 librados: starting msgr at 2024-03-25T18:14:37.866 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.861+0000 7f63b718c640 1 librados: starting objecter 2024-03-25T18:14:37.866 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.861+0000 7f63b718c640 1 Processor -- start 2024-03-25T18:14:37.866 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.861+0000 7f63b718c640 1 -- start start 2024-03-25T18:14:37.866 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.861+0000 7f63b718c640 1 librados: setting wanted keys 2024-03-25T18:14:37.866 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.861+0000 7f63b718c640 1 librados: calling monclient init 2024-03-25T18:14:37.866 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.861+0000 7f63b718c640 10 monclient: init 2024-03-25T18:14:37.866 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.861+0000 7f63b718c640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:37.866 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.861+0000 7f63b718c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:37.866 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.861+0000 7f63b718c640 10 monclient(hunting): picked mon.noname-b con 0x7f63b00a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:14:37.867 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.861+0000 7f63b718c640 10 monclient(hunting): picked mon.noname-c con 0x7f63b013f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:14:37.867 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.861+0000 7f63b718c640 10 monclient(hunting): picked mon.noname-a con 0x7f63b01430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:14:37.867 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.861+0000 7f63b718c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b00ab9c0 con 0x7f63b01430c0 2024-03-25T18:14:37.867 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.861+0000 7f63b718c640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b01d7810 con 0x7f63b00a98f0 2024-03-25T18:14:37.867 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.861+0000 7f63b718c640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b01d51f0 con 0x7f63b013f710 2024-03-25T18:14:37.867 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.861+0000 7f63b718c640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:37.867 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.861+0000 7f63b718c640 10 monclient(hunting): authenticate will time out at 7359.172748s 2024-03-25T18:14:37.872 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.871+0000 7f63b4f28640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f63b00a98f0 0x7f63b0138220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:46110/0 (socket says 172.21.15.156:46110) 2024-03-25T18:14:37.872 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.871+0000 7f63b4f28640 1 -- 172.21.15.156:0/608297867 learned_addr learned my addr 172.21.15.156:0/608297867 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:14:37.872 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.871+0000 7f63ad7fa640 10 client.?.objecter ms_handle_connect 0x7f63b00a98f0 2024-03-25T18:14:37.873 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.871+0000 7f63ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:37.876 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.875+0000 7f63ad7fa640 10 client.?.objecter ms_handle_connect 0x7f63b01430c0 2024-03-25T18:14:37.876 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.875+0000 7f63ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:37.878 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.877+0000 7f63ad7fa640 10 client.?.objecter ms_handle_connect 0x7f63b013f710 2024-03-25T18:14:37.879 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.877+0000 7f63ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:37.889 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.888+0000 7f63ad7fa640 1 -- 172.21.15.156:0/608297867 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3960893955 0 0) 0x7f63b00ab9c0 con 0x7f63b01430c0 2024-03-25T18:14:37.889 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.888+0000 7f63ad7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:14:37.890 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.888+0000 7f63ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:37.890 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.888+0000 7f63ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:37.890 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.888+0000 7f63ad7fa640 10 monclient(hunting): my global_id is 4506 2024-03-25T18:14:37.890 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.888+0000 7f63ad7fa640 1 -- 172.21.15.156:0/608297867 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6394004eb0 con 0x7f63b01430c0 2024-03-25T18:14:37.893 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.891+0000 7f63ad7fa640 1 -- 172.21.15.156:0/608297867 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1695406486 0 0) 0x7f63b01d7810 con 0x7f63b00a98f0 2024-03-25T18:14:37.893 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.891+0000 7f63ad7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:14:37.893 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.891+0000 7f63ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:37.893 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.891+0000 7f63ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:37.893 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.891+0000 7f63ad7fa640 10 monclient(hunting): my global_id is 4513 2024-03-25T18:14:37.893 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.891+0000 7f63ad7fa640 1 -- 172.21.15.156:0/608297867 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b00ab9c0 con 0x7f63b00a98f0 2024-03-25T18:14:37.893 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.891+0000 7f63ad7fa640 1 -- 172.21.15.156:0/608297867 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661782874 0 0) 0x7f63b01d51f0 con 0x7f63b013f710 2024-03-25T18:14:37.893 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.891+0000 7f63ad7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:14:37.893 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.891+0000 7f63ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:37.893 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.891+0000 7f63ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:37.894 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.891+0000 7f63ad7fa640 10 monclient(hunting): my global_id is 4517 2024-03-25T18:14:37.894 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.891+0000 7f63ad7fa640 1 -- 172.21.15.156:0/608297867 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b01d7810 con 0x7f63b013f710 2024-03-25T18:14:37.902 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.900+0000 7f63ad7fa640 1 -- 172.21.15.156:0/608297867 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3003275242 0 0) 0x7f6394004eb0 con 0x7f63b01430c0 2024-03-25T18:14:37.902 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.900+0000 7f63ad7fa640 1 -- 172.21.15.156:0/608297867 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63b01d51f0 con 0x7f63b01430c0 2024-03-25T18:14:37.902 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.900+0000 7f63ad7fa640 1 -- 172.21.15.156:0/608297867 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f63a8001ad0 con 0x7f63b01430c0 2024-03-25T18:14:37.902 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.900+0000 7f63ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:37.902 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.900+0000 7f63ad7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:37.902 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.900+0000 7f63ad7fa640 10 monclient(hunting): dump: 2024-03-25T18:14:37.902 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:37.902 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:37.903 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:37.903 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:37.903 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:37.903 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:37.903 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:37.903 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:37.903 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:37.903 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:37.903 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.900+0000 7f63ad7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:37.905 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.903+0000 7f63ad7fa640 1 -- 172.21.15.156:0/608297867 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1641845100 0 0) 0x7f63b01d7810 con 0x7f63b013f710 2024-03-25T18:14:37.905 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.903+0000 7f63ad7fa640 1 -- 172.21.15.156:0/608297867 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6394004eb0 con 0x7f63b013f710 2024-03-25T18:14:37.905 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.903+0000 7f63ad7fa640 1 -- 172.21.15.156:0/608297867 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f639c001860 con 0x7f63b013f710 2024-03-25T18:14:37.905 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.903+0000 7f63ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:37.905 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.903+0000 7f63ad7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:37.905 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.903+0000 7f63ad7fa640 10 monclient(hunting): dump: 2024-03-25T18:14:37.905 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:37.905 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:37.905 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:37.906 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:37.906 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:37.906 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:37.906 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:37.906 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:37.906 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:37.906 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:37.910 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.908+0000 7f63ad7fa640 1 -- 172.21.15.156:0/608297867 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2387061201 0 0) 0x7f63b01d51f0 con 0x7f63b01430c0 2024-03-25T18:14:37.910 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.908+0000 7f63ad7fa640 1 -- 172.21.15.156:0/608297867 >> v1:172.21.15.156:6790/0 conn(0x7f63b013f710 legacy=0x7f63b01d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:37.910 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.908+0000 7f63ad7fa640 1 -- 172.21.15.156:0/608297867 >> v1:172.21.15.194:6789/0 conn(0x7f63b00a98f0 legacy=0x7f63b0138220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:37.910 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.908+0000 7f63ad7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:14:37.910 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.908+0000 7f63ad7fa640 1 monclient: found mon.a 2024-03-25T18:14:37.910 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.908+0000 7f63ad7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:37.910 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.908+0000 7f63ad7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:14:37.910 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.908+0000 7f63ad7fa640 1 -- 172.21.15.156:0/608297867 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b0138930 con 0x7f63b01430c0 2024-03-25T18:14:37.910 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.908+0000 7f63ad7fa640 10 monclient: _finish_auth 0 2024-03-25T18:14:37.910 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.908+0000 7f63ad7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:14:37.911 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.908+0000 7f63ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:37.911 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.908+0000 7f63b718c640 5 monclient: authenticate success, global_id 4506 2024-03-25T18:14:37.911 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.908+0000 7f63b718c640 10 monclient: _renew_subs 2024-03-25T18:14:37.911 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.908+0000 7f63b718c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:14:37.911 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.908+0000 7f63b718c640 1 -- 172.21.15.156:0/608297867 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63b01d8440 con 0x7f63b01430c0 2024-03-25T18:14:37.911 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.908+0000 7f63b718c640 10 client.4506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:37.911 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.908+0000 7f63b718c640 10 monclient: _renew_subs 2024-03-25T18:14:37.911 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.908+0000 7f63b718c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:14:37.911 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.908+0000 7f63b718c640 1 -- 172.21.15.156:0/608297867 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63b0138df0 con 0x7f63b01430c0 2024-03-25T18:14:37.911 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.908+0000 7f63b718c640 1 librados: init done 2024-03-25T18:14:37.912 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.909+0000 7f63b718c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:37.912 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.909+0000 7f63b718c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:37.912 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.909+0000 7f63b718c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:14:37.912 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:37.909+0000 7f63b718c640 1 -- 172.21.15.156:0/608297867 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63b00627b0 con 0x7f63b01430c0 2024-03-25T18:14:38.241 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.239+0000 7f63ad7fa640 1 -- 172.21.15.156:0/608297867 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63a8001fe0 con 0x7f63b01430c0 2024-03-25T18:14:38.241 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.239+0000 7f63ad7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:38.241 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.239+0000 7f63ad7fa640 1 -- 172.21.15.156:0/608297867 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f63a8003b10 con 0x7f63b01430c0 2024-03-25T18:14:38.241 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.239+0000 7f63ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:38.241 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.239+0000 7f63ad7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:38.241 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.239+0000 7f63ad7fa640 10 monclient: dump: 2024-03-25T18:14:38.242 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:38.242 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:38.242 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:38.242 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:38.242 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:38.242 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:38.242 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:38.242 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:38.242 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:38.243 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:38.243 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.239+0000 7f63aeffd640 4 set_mon_vals no callback set 2024-03-25T18:14:38.243 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.240+0000 7f63ad7fa640 1 -- 172.21.15.156:0/608297867 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f63a8003d30 con 0x7f63b01430c0 2024-03-25T18:14:38.244 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.241+0000 7f63b4f28640 10 monclient: get_auth_request con 0x7f639406dec0 auth_method 0 2024-03-25T18:14:38.244 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.241+0000 7f63ad7fa640 10 client.4506.objecter ms_handle_connect 0x7f639406dec0 2024-03-25T18:14:38.250 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.248+0000 7f63ad7fa640 1 -- 172.21.15.156:0/608297867 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f63a8053760 con 0x7f63b01430c0 2024-03-25T18:14:38.250 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.248+0000 7f63ad7fa640 10 client.4506.objecter ms_dispatch 0x7f63b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:38.250 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.248+0000 7f63ad7fa640 3 client.4506.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:38.250 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.248+0000 7f63ad7fa640 3 client.4506.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:38.250 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.248+0000 7f63ad7fa640 20 client.4506.objecter dump_active .. 0 homeless 2024-03-25T18:14:38.250 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.248+0000 7f63ad7fa640 1 -- 172.21.15.156:0/608297867 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f63a804f270 con 0x7f63b01430c0 2024-03-25T18:14:38.250 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.248+0000 7f63ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:38.250 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.248+0000 7f63ad7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:38.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.506+0000 7f63b718c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:38.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.506+0000 7f63b718c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:38.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.506+0000 7f63b718c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:14:38.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.506+0000 7f63b718c640 1 -- 172.21.15.156:0/608297867 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63b0141a00 con 0x7f63b01430c0 2024-03-25T18:14:38.533 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.531+0000 7f63ad7fa640 1 -- 172.21.15.156:0/608297867 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f63a80511a0 con 0x7f63b01430c0 2024-03-25T18:14:38.533 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.531+0000 7f63ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:38.533 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.531+0000 7f63ad7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:38.533 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:14:38.533 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:38.536 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.534+0000 7f63b718c640 10 librados: watch_flush enter 2024-03-25T18:14:38.536 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.534+0000 7f63b718c640 10 librados: watch_flush exit 2024-03-25T18:14:38.536 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.534+0000 7f63b718c640 20 client.4506.objecter shutdown clearing up homeless session... 2024-03-25T18:14:38.536 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.534+0000 7f63b718c640 10 client.4506.objecter successfully canceled tick 2024-03-25T18:14:38.536 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.534+0000 7f63b718c640 1 -- 172.21.15.156:0/608297867 >> v1:172.21.15.194:6800/2946427713 conn(0x7f639406dec0 legacy=0x7f6394070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:38.536 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.534+0000 7f63b718c640 10 monclient: shutdown 2024-03-25T18:14:38.536 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.534+0000 7f63b718c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:38.536 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.534+0000 7f63b718c640 1 -- 172.21.15.156:0/608297867 >> v1:172.21.15.156:6789/0 conn(0x7f63b01430c0 legacy=0x7f63b01e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:38.536 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.534+0000 7f63b718c640 1 -- 172.21.15.156:0/608297867 shutdown_connections 2024-03-25T18:14:38.537 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.534+0000 7f63b718c640 1 -- 172.21.15.156:0/608297867 >> 172.21.15.156:0/608297867 conn(0x7f63b00b7ea0 msgr2=0x7f63b0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:38.537 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.534+0000 7f63b718c640 1 -- 172.21.15.156:0/608297867 shutdown_connections 2024-03-25T18:14:38.537 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.534+0000 7f63b718c640 1 -- 172.21.15.156:0/608297867 wait complete. 2024-03-25T18:14:38.537 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:38.534+0000 7f63b718c640 1 librados: shutdown 2024-03-25T18:14:38.548 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:45.354 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:45.466 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.465+0000 7f05c9b4d640 10 monclient: get_monmap_and_config 2024-03-25T18:14:45.466 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.465+0000 7f05c9b4d640 10 monclient: build_initial_monmap 2024-03-25T18:14:45.466 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.465+0000 7f05c9b4d640 10 monclient: monmap: 2024-03-25T18:14:45.466 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:14:45.466 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:45.466 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:14:45.466454+0000 2024-03-25T18:14:45.466 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:14:45.466454+0000 2024-03-25T18:14:45.467 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:45.467 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:45.467 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:14:45.467 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:14:45.467 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:14:45.467 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:45.468 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.466+0000 7f05c9b4d640 1 Processor -- start 2024-03-25T18:14:45.468 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.466+0000 7f05c9b4d640 1 -- start start 2024-03-25T18:14:45.468 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.466+0000 7f05c9b4d640 10 monclient: init 2024-03-25T18:14:45.468 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.467+0000 7f05c9b4d640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:45.468 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.467+0000 7f05c9b4d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:45.468 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.467+0000 7f05c9b4d640 10 monclient(hunting): picked mon.noname-a con 0x7f05c40b4d60 addr v1:172.21.15.156:6789/0 2024-03-25T18:14:45.468 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.467+0000 7f05c9b4d640 10 monclient(hunting): picked mon.noname-b con 0x7f05c40b1730 addr v1:172.21.15.156:6790/0 2024-03-25T18:14:45.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.467+0000 7f05c9b4d640 10 monclient(hunting): picked mon.noname-c con 0x7f05c414be90 addr v1:172.21.15.194:6789/0 2024-03-25T18:14:45.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.467+0000 7f05c9b4d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05c40af940 con 0x7f05c40b4d60 2024-03-25T18:14:45.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.467+0000 7f05c9b4d640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05c40b0a70 con 0x7f05c414be90 2024-03-25T18:14:45.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.467+0000 7f05c9b4d640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05c40b0d50 con 0x7f05c40b1730 2024-03-25T18:14:45.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.467+0000 7f05c9b4d640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:45.469 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.467+0000 7f05c9b4d640 10 monclient(hunting): authenticate will time out at 7366.778246s 2024-03-25T18:14:45.485 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.484+0000 7f05c3fff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f05c414be90 0x7f05c414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:54804/0 (socket says 172.21.15.156:54804) 2024-03-25T18:14:45.485 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.484+0000 7f05c3fff640 1 -- 172.21.15.156:0/389217727 learned_addr learned my addr 172.21.15.156:0/389217727 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:14:45.505 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.504+0000 7f05c27fc640 1 -- 172.21.15.156:0/389217727 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1081758058 0 0) 0x7f05c40b0a70 con 0x7f05c414be90 2024-03-25T18:14:45.505 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.504+0000 7f05c27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:45.505 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.504+0000 7f05c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:45.505 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.504+0000 7f05c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:45.505 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.504+0000 7f05c27fc640 10 monclient(hunting): my global_id is 4516 2024-03-25T18:14:45.505 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.504+0000 7f05c27fc640 1 -- 172.21.15.156:0/389217727 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05ac003520 con 0x7f05c414be90 2024-03-25T18:14:45.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.508+0000 7f05c27fc640 1 -- 172.21.15.156:0/389217727 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4250561007 0 0) 0x7f05c40b0d50 con 0x7f05c40b1730 2024-03-25T18:14:45.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.508+0000 7f05c27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:45.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.508+0000 7f05c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:45.510 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.508+0000 7f05c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:45.510 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.508+0000 7f05c27fc640 10 monclient(hunting): my global_id is 4520 2024-03-25T18:14:45.510 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.508+0000 7f05c27fc640 1 -- 172.21.15.156:0/389217727 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05c40b0a70 con 0x7f05c40b1730 2024-03-25T18:14:45.519 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.518+0000 7f05c27fc640 1 -- 172.21.15.156:0/389217727 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4233439104 0 0) 0x7f05c40af940 con 0x7f05c40b4d60 2024-03-25T18:14:45.519 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.518+0000 7f05c27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:45.519 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.518+0000 7f05c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:45.519 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.518+0000 7f05c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:45.519 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.518+0000 7f05c27fc640 10 monclient(hunting): my global_id is 4509 2024-03-25T18:14:45.519 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.518+0000 7f05c27fc640 1 -- 172.21.15.156:0/389217727 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05c40b0d50 con 0x7f05c40b4d60 2024-03-25T18:14:45.530 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.529+0000 7f05c27fc640 1 -- 172.21.15.156:0/389217727 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2572197881 0 0) 0x7f05ac003520 con 0x7f05c414be90 2024-03-25T18:14:45.530 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.529+0000 7f05c27fc640 1 -- 172.21.15.156:0/389217727 >> v1:172.21.15.156:6790/0 conn(0x7f05c40b1730 legacy=0x7f05c40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:45.530 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.529+0000 7f05c27fc640 1 -- 172.21.15.156:0/389217727 >> v1:172.21.15.156:6789/0 conn(0x7f05c40b4d60 legacy=0x7f05c413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:45.530 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.529+0000 7f05c27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:45.530 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.529+0000 7f05c27fc640 1 monclient: found mon.noname-c 2024-03-25T18:14:45.530 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.529+0000 7f05c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:45.530 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.529+0000 7f05c27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:14:45.531 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.529+0000 7f05c27fc640 1 -- 172.21.15.156:0/389217727 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c40b1030 con 0x7f05c414be90 2024-03-25T18:14:45.531 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.529+0000 7f05c27fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:45.531 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.529+0000 7f05c27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:45.531 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.529+0000 7f05c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:45.531 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.529+0000 7f05c27fc640 1 -- 172.21.15.156:0/389217727 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f05bc001a10 con 0x7f05c414be90 2024-03-25T18:14:45.531 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.529+0000 7f05c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:45.531 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.529+0000 7f05c27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:45.531 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.529+0000 7f05c27fc640 10 monclient: dump: 2024-03-25T18:14:45.531 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:45.531 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:45.531 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:45.531 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:45.531 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:45.531 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:45.532 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:45.532 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:45.532 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:45.532 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:45.532 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.529+0000 7f05c9b4d640 5 monclient: authenticate success, global_id 4516 2024-03-25T18:14:45.532 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.529+0000 7f05c9b4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:45.561 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.560+0000 7f05c27fc640 1 -- 172.21.15.156:0/389217727 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05bc001b70 con 0x7f05c414be90 2024-03-25T18:14:45.561 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.560+0000 7f05c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:45.561 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.560+0000 7f05c27fc640 1 -- 172.21.15.156:0/389217727 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f05bc001f80 con 0x7f05c414be90 2024-03-25T18:14:45.562 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.560+0000 7f05c9b4d640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:45.562 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.560+0000 7f05c9b4d640 4 set_mon_vals no callback set 2024-03-25T18:14:45.562 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.560+0000 7f05c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:45.562 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.560+0000 7f05c27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:45.562 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.560+0000 7f05c27fc640 10 monclient: dump: 2024-03-25T18:14:45.562 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:45.562 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:45.562 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:45.562 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:45.562 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:45.562 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:45.562 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:45.562 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:45.562 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:45.563 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:45.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.561+0000 7f05c9b4d640 10 monclient: shutdown 2024-03-25T18:14:45.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.561+0000 7f05c9b4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:45.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.561+0000 7f05c9b4d640 1 -- 172.21.15.156:0/389217727 >> v1:172.21.15.194:6789/0 conn(0x7f05c414be90 legacy=0x7f05c414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:45.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.561+0000 7f05c9b4d640 1 -- 172.21.15.156:0/389217727 shutdown_connections 2024-03-25T18:14:45.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.561+0000 7f05c9b4d640 1 -- 172.21.15.156:0/389217727 >> 172.21.15.156:0/389217727 conn(0x7f05c40617a0 msgr2=0x7f05c4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:45.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.561+0000 7f05c9b4d640 1 -- 172.21.15.156:0/389217727 shutdown_connections 2024-03-25T18:14:45.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.561+0000 7f05c9b4d640 1 -- 172.21.15.156:0/389217727 wait complete. 2024-03-25T18:14:45.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.562+0000 7f05c9b4d640 10 monclient: build_initial_monmap 2024-03-25T18:14:45.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.562+0000 7f05c9b4d640 10 monclient: monmap: 2024-03-25T18:14:45.565 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:14:45.565 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:45.566 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:14:45.566 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:14:45.566 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:45.566 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:14:45.566 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:14:45.566 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:14:45.566 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:14:45.566 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:45.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.562+0000 7f05c9b4d640 1 librados: starting msgr at 2024-03-25T18:14:45.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.562+0000 7f05c9b4d640 1 librados: starting objecter 2024-03-25T18:14:45.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.562+0000 7f05c9b4d640 1 Processor -- start 2024-03-25T18:14:45.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.563+0000 7f05c9b4d640 1 -- start start 2024-03-25T18:14:45.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.563+0000 7f05c9b4d640 1 librados: setting wanted keys 2024-03-25T18:14:45.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.563+0000 7f05c9b4d640 1 librados: calling monclient init 2024-03-25T18:14:45.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.563+0000 7f05c9b4d640 10 monclient: init 2024-03-25T18:14:45.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.563+0000 7f05c9b4d640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:45.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.563+0000 7f05c9b4d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:45.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.564+0000 7f05c9b4d640 10 monclient(hunting): picked mon.noname-c con 0x7f05c40b1730 addr v1:172.21.15.156:6790/0 2024-03-25T18:14:45.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.564+0000 7f05c9b4d640 10 monclient(hunting): picked mon.noname-b con 0x7f05c40b4d60 addr v1:172.21.15.194:6789/0 2024-03-25T18:14:45.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.564+0000 7f05c9b4d640 10 monclient(hunting): picked mon.noname-a con 0x7f05c414be90 addr v1:172.21.15.156:6789/0 2024-03-25T18:14:45.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.564+0000 7f05c9b4d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05c41dad20 con 0x7f05c414be90 2024-03-25T18:14:45.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.564+0000 7f05c9b4d640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05c4138770 con 0x7f05c40b4d60 2024-03-25T18:14:45.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.564+0000 7f05c9b4d640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05c4137450 con 0x7f05c40b1730 2024-03-25T18:14:45.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.564+0000 7f05c9b4d640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:45.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.564+0000 7f05c9b4d640 10 monclient(hunting): authenticate will time out at 7366.874944s 2024-03-25T18:14:45.569 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.568+0000 7f05c37fe640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f05c40b1730 0x7f05c4136d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:57932/0 (socket says 172.21.15.156:57932) 2024-03-25T18:14:45.569 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.568+0000 7f05c37fe640 1 -- 172.21.15.156:0/2557619307 learned_addr learned my addr 172.21.15.156:0/2557619307 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:14:45.571 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.570+0000 7f05abfff640 10 client.?.objecter ms_handle_connect 0x7f05c40b1730 2024-03-25T18:14:45.571 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.570+0000 7f05abfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:45.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.578+0000 7f05abfff640 10 client.?.objecter ms_handle_connect 0x7f05c414be90 2024-03-25T18:14:45.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.578+0000 7f05abfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:45.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.579+0000 7f05abfff640 1 -- 172.21.15.156:0/2557619307 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479823720 0 0) 0x7f05c4137450 con 0x7f05c40b1730 2024-03-25T18:14:45.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.579+0000 7f05abfff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:45.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.579+0000 7f05abfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:45.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.579+0000 7f05abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:45.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.579+0000 7f05abfff640 10 monclient(hunting): my global_id is 4523 2024-03-25T18:14:45.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.579+0000 7f05abfff640 1 -- 172.21.15.156:0/2557619307 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05b0004eb0 con 0x7f05c40b1730 2024-03-25T18:14:45.584 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.583+0000 7f05abfff640 10 client.?.objecter ms_handle_connect 0x7f05c40b4d60 2024-03-25T18:14:45.584 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.583+0000 7f05abfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:45.588 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.587+0000 7f05abfff640 1 -- 172.21.15.156:0/2557619307 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500053682 0 0) 0x7f05c4138770 con 0x7f05c40b4d60 2024-03-25T18:14:45.588 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.587+0000 7f05abfff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:45.588 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.587+0000 7f05abfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:45.588 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.587+0000 7f05abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:45.588 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.587+0000 7f05abfff640 10 monclient(hunting): my global_id is 4519 2024-03-25T18:14:45.588 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.587+0000 7f05abfff640 1 -- 172.21.15.156:0/2557619307 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05c4137450 con 0x7f05c40b4d60 2024-03-25T18:14:45.602 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.600+0000 7f05abfff640 1 -- 172.21.15.156:0/2557619307 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3081264633 0 0) 0x7f05b0004eb0 con 0x7f05c40b1730 2024-03-25T18:14:45.602 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.600+0000 7f05abfff640 1 -- 172.21.15.156:0/2557619307 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05c4138770 con 0x7f05c40b1730 2024-03-25T18:14:45.602 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.600+0000 7f05abfff640 1 -- 172.21.15.156:0/2557619307 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f05b4001a10 con 0x7f05c40b1730 2024-03-25T18:14:45.602 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.600+0000 7f05abfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:45.602 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.600+0000 7f05abfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:45.602 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.600+0000 7f05abfff640 10 monclient(hunting): dump: 2024-03-25T18:14:45.602 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:45.602 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:45.602 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:45.602 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:45.602 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:45.603 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:45.603 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:45.603 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:45.603 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:45.603 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:45.603 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.600+0000 7f05abfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:45.603 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.600+0000 7f05abfff640 1 -- 172.21.15.156:0/2557619307 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1645167142 0 0) 0x7f05c41dad20 con 0x7f05c414be90 2024-03-25T18:14:45.603 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.600+0000 7f05abfff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:45.603 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.600+0000 7f05abfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:45.603 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.600+0000 7f05abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:45.603 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.600+0000 7f05abfff640 10 monclient(hunting): my global_id is 4512 2024-03-25T18:14:45.603 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.600+0000 7f05abfff640 1 -- 172.21.15.156:0/2557619307 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05b0004eb0 con 0x7f05c414be90 2024-03-25T18:14:45.610 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.609+0000 7f05abfff640 1 -- 172.21.15.156:0/2557619307 <== mon.2 v1:172.21.15.156:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 816672324 0 0) 0x7f05c4138770 con 0x7f05c40b1730 2024-03-25T18:14:45.610 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.609+0000 7f05abfff640 1 -- 172.21.15.156:0/2557619307 >> v1:172.21.15.194:6789/0 conn(0x7f05c40b4d60 legacy=0x7f05c41db1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:45.610 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.609+0000 7f05abfff640 1 -- 172.21.15.156:0/2557619307 >> v1:172.21.15.156:6789/0 conn(0x7f05c414be90 legacy=0x7f05c412fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:45.611 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.609+0000 7f05abfff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:45.611 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.609+0000 7f05abfff640 1 monclient: found mon.c 2024-03-25T18:14:45.611 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.609+0000 7f05abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:45.611 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.609+0000 7f05abfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:14:45.611 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.609+0000 7f05abfff640 1 -- 172.21.15.156:0/2557619307 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c41375e0 con 0x7f05c40b1730 2024-03-25T18:14:45.611 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.609+0000 7f05abfff640 10 monclient: _finish_auth 0 2024-03-25T18:14:45.611 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.609+0000 7f05abfff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:45.611 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.609+0000 7f05abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:45.611 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.609+0000 7f05c9b4d640 5 monclient: authenticate success, global_id 4523 2024-03-25T18:14:45.611 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.609+0000 7f05c9b4d640 10 monclient: _renew_subs 2024-03-25T18:14:45.611 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.609+0000 7f05c9b4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:14:45.611 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.609+0000 7f05c9b4d640 1 -- 172.21.15.156:0/2557619307 --> v1:172.21.15.156:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05c4143900 con 0x7f05c40b1730 2024-03-25T18:14:45.611 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.609+0000 7f05c9b4d640 10 client.4523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:45.611 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.609+0000 7f05c9b4d640 10 monclient: _renew_subs 2024-03-25T18:14:45.611 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.609+0000 7f05c9b4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:14:45.612 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.609+0000 7f05c9b4d640 1 -- 172.21.15.156:0/2557619307 --> v1:172.21.15.156:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c41dbaa0 con 0x7f05c40b1730 2024-03-25T18:14:45.612 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.609+0000 7f05c9b4d640 1 librados: init done 2024-03-25T18:14:45.612 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.610+0000 7f05c9b4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:45.612 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.610+0000 7f05c9b4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:45.612 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.610+0000 7f05c9b4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:14:45.612 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.610+0000 7f05c9b4d640 1 -- 172.21.15.156:0/2557619307 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05c40627b0 con 0x7f05c40b1730 2024-03-25T18:14:45.913 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.911+0000 7f05abfff640 1 -- 172.21.15.156:0/2557619307 <== mon.2 v1:172.21.15.156:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05b4001c70 con 0x7f05c40b1730 2024-03-25T18:14:45.913 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.911+0000 7f05abfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:45.913 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.911+0000 7f05abfff640 1 -- 172.21.15.156:0/2557619307 <== mon.2 v1:172.21.15.156:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f05b4003a60 con 0x7f05c40b1730 2024-03-25T18:14:45.913 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.911+0000 7f05abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:45.913 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.911+0000 7f05abfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:45.913 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.911+0000 7f05abfff640 10 monclient: dump: 2024-03-25T18:14:45.913 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:45.913 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:45.914 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:45.914 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:45.914 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:45.914 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:45.914 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:45.914 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:45.914 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:45.914 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:45.914 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.911+0000 7f05c1ffb640 4 set_mon_vals no callback set 2024-03-25T18:14:45.914 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.912+0000 7f05abfff640 1 -- 172.21.15.156:0/2557619307 <== mon.2 v1:172.21.15.156:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f05b4003c80 con 0x7f05c40b1730 2024-03-25T18:14:45.915 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.912+0000 7f05c2ffd640 10 monclient: get_auth_request con 0x7f05b006de00 auth_method 0 2024-03-25T18:14:45.915 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.913+0000 7f05abfff640 10 client.4523.objecter ms_handle_connect 0x7f05b006de00 2024-03-25T18:14:45.920 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.919+0000 7f05abfff640 1 -- 172.21.15.156:0/2557619307 <== mon.2 v1:172.21.15.156:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f05b4054860 con 0x7f05c40b1730 2024-03-25T18:14:45.920 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.919+0000 7f05abfff640 10 client.4523.objecter ms_dispatch 0x7f05c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:45.920 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.919+0000 7f05abfff640 3 client.4523.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:45.920 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.919+0000 7f05abfff640 3 client.4523.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:45.920 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.919+0000 7f05abfff640 20 client.4523.objecter dump_active .. 0 homeless 2024-03-25T18:14:45.920 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.919+0000 7f05abfff640 1 -- 172.21.15.156:0/2557619307 <== mon.2 v1:172.21.15.156:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f05b4050370 con 0x7f05c40b1730 2024-03-25T18:14:45.920 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.919+0000 7f05abfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:45.920 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:45.919+0000 7f05abfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:46.171 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:46.169+0000 7f05c9b4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:46.171 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:46.169+0000 7f05c9b4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:46.171 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:46.170+0000 7f05c9b4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:14:46.171 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:46.170+0000 7f05c9b4d640 1 -- 172.21.15.156:0/2557619307 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05c4069f40 con 0x7f05c40b1730 2024-03-25T18:14:46.195 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:46.194+0000 7f05abfff640 1 -- 172.21.15.156:0/2557619307 <== mon.2 v1:172.21.15.156:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f05b40522a0 con 0x7f05c40b1730 2024-03-25T18:14:46.195 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:46.194+0000 7f05abfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:46.195 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:46.194+0000 7f05abfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:46.196 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:14:46.196 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:46.198 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:46.196+0000 7f05c9b4d640 10 librados: watch_flush enter 2024-03-25T18:14:46.198 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:46.196+0000 7f05c9b4d640 10 librados: watch_flush exit 2024-03-25T18:14:46.198 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:46.196+0000 7f05c9b4d640 20 client.4523.objecter shutdown clearing up homeless session... 2024-03-25T18:14:46.198 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:46.196+0000 7f05c9b4d640 10 client.4523.objecter successfully canceled tick 2024-03-25T18:14:46.199 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:46.197+0000 7f05c9b4d640 1 -- 172.21.15.156:0/2557619307 >> v1:172.21.15.194:6800/2946427713 conn(0x7f05b006de00 legacy=0x7f05b00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:46.199 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:46.197+0000 7f05c9b4d640 10 monclient: shutdown 2024-03-25T18:14:46.199 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:46.197+0000 7f05c9b4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:46.199 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:46.197+0000 7f05c9b4d640 1 -- 172.21.15.156:0/2557619307 >> v1:172.21.15.156:6790/0 conn(0x7f05c40b1730 legacy=0x7f05c4136d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:46.199 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:46.197+0000 7f05c9b4d640 1 -- 172.21.15.156:0/2557619307 shutdown_connections 2024-03-25T18:14:46.199 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:46.197+0000 7f05c9b4d640 1 -- 172.21.15.156:0/2557619307 >> 172.21.15.156:0/2557619307 conn(0x7f05c40617a0 msgr2=0x7f05c40b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:46.199 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:46.197+0000 7f05c9b4d640 1 -- 172.21.15.156:0/2557619307 shutdown_connections 2024-03-25T18:14:46.199 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:46.197+0000 7f05c9b4d640 1 -- 172.21.15.156:0/2557619307 wait complete. 2024-03-25T18:14:46.199 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:46.197+0000 7f05c9b4d640 1 librados: shutdown 2024-03-25T18:14:46.211 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:49.297 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:06:55.287 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:49.348 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:06:55.338 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:49.392 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:06:55.382 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:49.398 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:06:55.389 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:49.402 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:06:55.369 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:49.412 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:06:55.379 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:49.416 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:06:55.406 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:49.427 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:06:55.394 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:49.444 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:06:55.434 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:53.018 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:53.130 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.129+0000 7f80a8158640 10 monclient: get_monmap_and_config 2024-03-25T18:14:53.130 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.129+0000 7f80a8158640 10 monclient: build_initial_monmap 2024-03-25T18:14:53.130 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.129+0000 7f80a8158640 10 monclient: monmap: 2024-03-25T18:14:53.131 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:14:53.131 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:53.131 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:14:53.130660+0000 2024-03-25T18:14:53.131 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:14:53.130660+0000 2024-03-25T18:14:53.131 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:53.131 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:53.131 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:14:53.131 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:14:53.131 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:14:53.131 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:53.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.130+0000 7f80a8158640 1 Processor -- start 2024-03-25T18:14:53.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.130+0000 7f80a8158640 1 -- start start 2024-03-25T18:14:53.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.130+0000 7f80a8158640 10 monclient: init 2024-03-25T18:14:53.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.131+0000 7f80a8158640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:53.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.131+0000 7f80a8158640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:53.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.131+0000 7f80a8158640 10 monclient(hunting): picked mon.noname-b con 0x7f80a00a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:14:53.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.131+0000 7f80a8158640 10 monclient(hunting): picked mon.noname-c con 0x7f80a013f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:14:53.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.132+0000 7f80a8158640 10 monclient(hunting): picked mon.noname-a con 0x7f80a01430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:14:53.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.132+0000 7f80a8158640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a0142520 con 0x7f80a01430c0 2024-03-25T18:14:53.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.132+0000 7f80a8158640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a00b28a0 con 0x7f80a013f710 2024-03-25T18:14:53.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.132+0000 7f80a8158640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a00b2a30 con 0x7f80a00a98f0 2024-03-25T18:14:53.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.132+0000 7f80a8158640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:53.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.132+0000 7f80a8158640 10 monclient(hunting): authenticate will time out at 7374.443239s 2024-03-25T18:14:53.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.143+0000 7f80a56f3640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f80a013f710 0x7f80a01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:54372/0 (socket says 172.21.15.156:54372) 2024-03-25T18:14:53.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.143+0000 7f80a56f3640 1 -- 172.21.15.156:0/1880512956 learned_addr learned my addr 172.21.15.156:0/1880512956 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:14:53.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.165+0000 7f80a4ef2640 1 -- 172.21.15.156:0/1880512956 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055257436 0 0) 0x7f80a00b28a0 con 0x7f80a013f710 2024-03-25T18:14:53.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.165+0000 7f80a4ef2640 10 monclient(hunting): _negotiate 2024-03-25T18:14:53.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.165+0000 7f80a4ef2640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:53.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.165+0000 7f80a4ef2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:53.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.165+0000 7f80a4ef2640 10 monclient(hunting): my global_id is 4522 2024-03-25T18:14:53.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.165+0000 7f80a4ef2640 1 -- 172.21.15.156:0/1880512956 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f808c003520 con 0x7f80a013f710 2024-03-25T18:14:53.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.166+0000 7f80a4ef2640 1 -- 172.21.15.156:0/1880512956 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366029072 0 0) 0x7f80a00b2a30 con 0x7f80a00a98f0 2024-03-25T18:14:53.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.166+0000 7f80a4ef2640 10 monclient(hunting): _negotiate 2024-03-25T18:14:53.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.166+0000 7f80a4ef2640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:53.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.166+0000 7f80a4ef2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:53.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.166+0000 7f80a4ef2640 10 monclient(hunting): my global_id is 4526 2024-03-25T18:14:53.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.166+0000 7f80a4ef2640 1 -- 172.21.15.156:0/1880512956 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80a00b28a0 con 0x7f80a00a98f0 2024-03-25T18:14:53.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.177+0000 7f80a4ef2640 1 -- 172.21.15.156:0/1880512956 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4172457599 0 0) 0x7f80a0142520 con 0x7f80a01430c0 2024-03-25T18:14:53.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.177+0000 7f80a4ef2640 10 monclient(hunting): _negotiate 2024-03-25T18:14:53.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.177+0000 7f80a4ef2640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:53.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.177+0000 7f80a4ef2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:53.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.177+0000 7f80a4ef2640 10 monclient(hunting): my global_id is 4515 2024-03-25T18:14:53.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.177+0000 7f80a4ef2640 1 -- 172.21.15.156:0/1880512956 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80a00b2a30 con 0x7f80a01430c0 2024-03-25T18:14:53.182 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.180+0000 7f80a4ef2640 1 -- 172.21.15.156:0/1880512956 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3311262794 0 0) 0x7f80a00b28a0 con 0x7f80a00a98f0 2024-03-25T18:14:53.182 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.181+0000 7f80a4ef2640 1 -- 172.21.15.156:0/1880512956 >> v1:172.21.15.194:6789/0 conn(0x7f80a013f710 legacy=0x7f80a01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:53.182 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.181+0000 7f80a4ef2640 1 -- 172.21.15.156:0/1880512956 >> v1:172.21.15.156:6789/0 conn(0x7f80a01430c0 legacy=0x7f80a014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:53.182 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.181+0000 7f80a4ef2640 10 monclient: _finish_hunting 0 2024-03-25T18:14:53.182 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.181+0000 7f80a4ef2640 1 monclient: found mon.noname-b 2024-03-25T18:14:53.182 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.181+0000 7f80a4ef2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:53.182 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.181+0000 7f80a4ef2640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.156:6790/0 2024-03-25T18:14:53.182 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.181+0000 7f80a4ef2640 1 -- 172.21.15.156:0/1880512956 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a00b44c0 con 0x7f80a00a98f0 2024-03-25T18:14:53.182 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.181+0000 7f80a4ef2640 10 monclient: _finish_auth 0 2024-03-25T18:14:53.182 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.181+0000 7f80a4ef2640 10 monclient: _check_auth_tickets 2024-03-25T18:14:53.182 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.181+0000 7f80a4ef2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:53.182 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.181+0000 7f80a4ef2640 1 -- 172.21.15.156:0/1880512956 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f809c001670 con 0x7f80a00a98f0 2024-03-25T18:14:53.183 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.181+0000 7f80a8158640 5 monclient: authenticate success, global_id 4526 2024-03-25T18:14:53.183 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.181+0000 7f80a8158640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:53.183 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.181+0000 7f80a4ef2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:53.183 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.181+0000 7f80a4ef2640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:53.183 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.181+0000 7f80a4ef2640 10 monclient: dump: 2024-03-25T18:14:53.183 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:53.183 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:53.183 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:53.183 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:53.183 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:53.183 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:53.183 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:53.183 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:53.183 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:53.183 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:53.183 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.181+0000 7f80a8158640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:53.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.198+0000 7f80a4ef2640 1 -- 172.21.15.156:0/1880512956 <== mon.2 v1:172.21.15.156:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f809c0017d0 con 0x7f80a00a98f0 2024-03-25T18:14:53.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.198+0000 7f80a4ef2640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:53.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.198+0000 7f80a4ef2640 1 -- 172.21.15.156:0/1880512956 <== mon.2 v1:172.21.15.156:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f809c001be0 con 0x7f80a00a98f0 2024-03-25T18:14:53.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.198+0000 7f80a4ef2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:53.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.198+0000 7f80a4ef2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:53.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.198+0000 7f80a4ef2640 10 monclient: dump: 2024-03-25T18:14:53.200 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:53.200 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:53.200 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:53.200 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:53.200 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:53.200 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:53.201 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:53.201 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:53.201 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:53.201 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:53.201 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.198+0000 7f80a8158640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:53.201 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.198+0000 7f80a8158640 4 set_mon_vals no callback set 2024-03-25T18:14:53.202 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.199+0000 7f80a8158640 10 monclient: shutdown 2024-03-25T18:14:53.202 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.199+0000 7f80a8158640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:53.202 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.199+0000 7f80a8158640 1 -- 172.21.15.156:0/1880512956 >> v1:172.21.15.156:6790/0 conn(0x7f80a00a98f0 legacy=0x7f80a00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:53.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.199+0000 7f80a8158640 1 -- 172.21.15.156:0/1880512956 shutdown_connections 2024-03-25T18:14:53.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.199+0000 7f80a8158640 1 -- 172.21.15.156:0/1880512956 >> 172.21.15.156:0/1880512956 conn(0x7f80a00b7ea0 msgr2=0x7f80a00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:53.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.200+0000 7f80a8158640 1 -- 172.21.15.156:0/1880512956 shutdown_connections 2024-03-25T18:14:53.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.200+0000 7f80a8158640 1 -- 172.21.15.156:0/1880512956 wait complete. 2024-03-25T18:14:53.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.200+0000 7f80a8158640 10 monclient: build_initial_monmap 2024-03-25T18:14:53.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.201+0000 7f80a8158640 10 monclient: monmap: 2024-03-25T18:14:53.203 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:14:53.203 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:53.203 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:14:53.203 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:14:53.203 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:53.203 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:14:53.203 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:14:53.204 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:14:53.204 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:14:53.204 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:53.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.201+0000 7f80a8158640 1 librados: starting msgr at 2024-03-25T18:14:53.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.201+0000 7f80a8158640 1 librados: starting objecter 2024-03-25T18:14:53.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.201+0000 7f80a8158640 1 Processor -- start 2024-03-25T18:14:53.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.201+0000 7f80a8158640 1 -- start start 2024-03-25T18:14:53.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.201+0000 7f80a8158640 1 librados: setting wanted keys 2024-03-25T18:14:53.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.201+0000 7f80a8158640 1 librados: calling monclient init 2024-03-25T18:14:53.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.201+0000 7f80a8158640 10 monclient: init 2024-03-25T18:14:53.205 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.202+0000 7f80a8158640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:53.205 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.202+0000 7f80a8158640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:53.205 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.202+0000 7f80a8158640 10 monclient(hunting): picked mon.noname-b con 0x7f80a00a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:14:53.205 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.202+0000 7f80a8158640 10 monclient(hunting): picked mon.noname-c con 0x7f80a013f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:14:53.206 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.202+0000 7f80a8158640 10 monclient(hunting): picked mon.noname-a con 0x7f80a01430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:14:53.206 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.202+0000 7f80a8158640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a0136e60 con 0x7f80a01430c0 2024-03-25T18:14:53.206 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.202+0000 7f80a8158640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a01381a0 con 0x7f80a00a98f0 2024-03-25T18:14:53.206 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.202+0000 7f80a8158640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a01370b0 con 0x7f80a013f710 2024-03-25T18:14:53.206 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.202+0000 7f80a8158640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:53.206 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.202+0000 7f80a8158640 10 monclient(hunting): authenticate will time out at 7374.513573s 2024-03-25T18:14:53.212 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.211+0000 7f80a5ef4640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f80a00a98f0 0x7f80a0137a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:54410/0 (socket says 172.21.15.156:54410) 2024-03-25T18:14:53.212 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.211+0000 7f80a5ef4640 1 -- 172.21.15.156:0/546983466 learned_addr learned my addr 172.21.15.156:0/546983466 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:14:53.216 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.215+0000 7f80967fc640 10 client.?.objecter ms_handle_connect 0x7f80a00a98f0 2024-03-25T18:14:53.216 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.215+0000 7f80967fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:53.220 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.219+0000 7f80967fc640 10 client.?.objecter ms_handle_connect 0x7f80a013f710 2024-03-25T18:14:53.220 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.219+0000 7f80967fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:53.222 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.221+0000 7f80967fc640 10 client.?.objecter ms_handle_connect 0x7f80a01430c0 2024-03-25T18:14:53.222 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.221+0000 7f80967fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:53.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.228+0000 7f80967fc640 1 -- 172.21.15.156:0/546983466 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873741766 0 0) 0x7f80a0136e60 con 0x7f80a01430c0 2024-03-25T18:14:53.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.228+0000 7f80967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:53.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.228+0000 7f80967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:53.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.228+0000 7f80967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:53.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.228+0000 7f80967fc640 10 monclient(hunting): my global_id is 4518 2024-03-25T18:14:53.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.228+0000 7f80967fc640 1 -- 172.21.15.156:0/546983466 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8084004eb0 con 0x7f80a01430c0 2024-03-25T18:14:53.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.229+0000 7f80967fc640 1 -- 172.21.15.156:0/546983466 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032497426 0 0) 0x7f80a01370b0 con 0x7f80a013f710 2024-03-25T18:14:53.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.229+0000 7f80967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:53.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.229+0000 7f80967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:53.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.229+0000 7f80967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:53.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.229+0000 7f80967fc640 10 monclient(hunting): my global_id is 4529 2024-03-25T18:14:53.231 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.229+0000 7f80967fc640 1 -- 172.21.15.156:0/546983466 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80a0136e60 con 0x7f80a013f710 2024-03-25T18:14:53.236 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.235+0000 7f80967fc640 1 -- 172.21.15.156:0/546983466 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1670141378 0 0) 0x7f80a01381a0 con 0x7f80a00a98f0 2024-03-25T18:14:53.236 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.235+0000 7f80967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:53.236 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.235+0000 7f80967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:53.236 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.235+0000 7f80967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:53.236 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.235+0000 7f80967fc640 10 monclient(hunting): my global_id is 4525 2024-03-25T18:14:53.236 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.235+0000 7f80967fc640 1 -- 172.21.15.156:0/546983466 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80a01370b0 con 0x7f80a00a98f0 2024-03-25T18:14:53.241 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.240+0000 7f80967fc640 1 -- 172.21.15.156:0/546983466 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2333785439 0 0) 0x7f8084004eb0 con 0x7f80a01430c0 2024-03-25T18:14:53.241 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.240+0000 7f80967fc640 1 -- 172.21.15.156:0/546983466 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80a01381a0 con 0x7f80a01430c0 2024-03-25T18:14:53.241 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.240+0000 7f80967fc640 1 -- 172.21.15.156:0/546983466 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f8098001bd0 con 0x7f80a01430c0 2024-03-25T18:14:53.242 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.240+0000 7f80967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:53.242 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.240+0000 7f80967fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:53.242 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.240+0000 7f80967fc640 10 monclient(hunting): dump: 2024-03-25T18:14:53.242 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:53.242 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:53.242 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:53.242 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:53.242 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:53.242 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:53.242 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:53.242 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:53.242 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:53.243 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:53.243 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.240+0000 7f80967fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:53.253 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.251+0000 7f80967fc640 1 -- 172.21.15.156:0/546983466 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4142316463 0 0) 0x7f80a01381a0 con 0x7f80a01430c0 2024-03-25T18:14:53.253 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.251+0000 7f80967fc640 1 -- 172.21.15.156:0/546983466 >> v1:172.21.15.156:6790/0 conn(0x7f80a013f710 legacy=0x7f80a013a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:53.253 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.251+0000 7f80967fc640 1 -- 172.21.15.156:0/546983466 >> v1:172.21.15.194:6789/0 conn(0x7f80a00a98f0 legacy=0x7f80a0137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:53.253 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.251+0000 7f80967fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:53.253 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.251+0000 7f80967fc640 1 monclient: found mon.a 2024-03-25T18:14:53.253 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.251+0000 7f80967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:53.253 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.251+0000 7f80967fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:14:53.253 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.251+0000 7f80967fc640 1 -- 172.21.15.156:0/546983466 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a0137390 con 0x7f80a01430c0 2024-03-25T18:14:53.253 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.251+0000 7f80967fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:53.253 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.251+0000 7f80967fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:53.253 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.251+0000 7f80967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:53.253 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.251+0000 7f80a8158640 5 monclient: authenticate success, global_id 4518 2024-03-25T18:14:53.253 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.251+0000 7f80a8158640 10 monclient: _renew_subs 2024-03-25T18:14:53.254 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.251+0000 7f80a8158640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:14:53.254 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.251+0000 7f80a8158640 1 -- 172.21.15.156:0/546983466 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a01376d0 con 0x7f80a01430c0 2024-03-25T18:14:53.254 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.251+0000 7f80a8158640 10 client.4518.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:53.254 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.251+0000 7f80a8158640 10 monclient: _renew_subs 2024-03-25T18:14:53.254 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.251+0000 7f80a8158640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:14:53.254 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.251+0000 7f80a8158640 1 -- 172.21.15.156:0/546983466 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a0135530 con 0x7f80a01430c0 2024-03-25T18:14:53.254 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.251+0000 7f80a8158640 1 librados: init done 2024-03-25T18:14:53.255 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.252+0000 7f80a8158640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:53.255 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.252+0000 7f80a8158640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:53.255 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.252+0000 7f80a8158640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:14:53.255 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.252+0000 7f80a8158640 1 -- 172.21.15.156:0/546983466 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a00b3ef0 con 0x7f80a01430c0 2024-03-25T18:14:53.287 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.286+0000 7f80967fc640 1 -- 172.21.15.156:0/546983466 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8098002060 con 0x7f80a01430c0 2024-03-25T18:14:53.287 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.286+0000 7f80967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:53.287 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.286+0000 7f80967fc640 1 -- 172.21.15.156:0/546983466 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f8098003bf0 con 0x7f80a01430c0 2024-03-25T18:14:53.287 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.286+0000 7f80967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:53.287 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.286+0000 7f80967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:53.287 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.286+0000 7f80967fc640 10 monclient: dump: 2024-03-25T18:14:53.288 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:14:53.288 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:14:53.288 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:53.288 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:14:53.288 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:53.288 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:14:53.288 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:14:53.288 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:14:53.288 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:14:53.288 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:14:53.288 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.286+0000 7f8097fff640 4 set_mon_vals no callback set 2024-03-25T18:14:53.289 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.287+0000 7f80967fc640 1 -- 172.21.15.156:0/546983466 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f8098003e10 con 0x7f80a01430c0 2024-03-25T18:14:53.289 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.288+0000 7f80a5ef4640 10 monclient: get_auth_request con 0x7f808406ddf0 auth_method 0 2024-03-25T18:14:53.289 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.288+0000 7f80967fc640 10 client.4518.objecter ms_handle_connect 0x7f808406ddf0 2024-03-25T18:14:53.291 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.290+0000 7f80967fc640 1 -- 172.21.15.156:0/546983466 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f8098053890 con 0x7f80a01430c0 2024-03-25T18:14:53.291 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.290+0000 7f80967fc640 10 client.4518.objecter ms_dispatch 0x7f80a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:53.291 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.290+0000 7f80967fc640 3 client.4518.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:53.291 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.290+0000 7f80967fc640 3 client.4518.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:53.291 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.290+0000 7f80967fc640 20 client.4518.objecter dump_active .. 0 homeless 2024-03-25T18:14:53.529 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.528+0000 7f80967fc640 1 -- 172.21.15.156:0/546983466 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f809804f3a0 con 0x7f80a01430c0 2024-03-25T18:14:53.529 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.528+0000 7f80967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:53.529 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.528+0000 7f80967fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:53.772 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.770+0000 7f80a8158640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:53.772 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.771+0000 7f80a8158640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:53.772 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.771+0000 7f80a8158640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:14:53.772 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.771+0000 7f80a8158640 1 -- 172.21.15.156:0/546983466 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80a0137520 con 0x7f80a01430c0 2024-03-25T18:14:53.793 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.792+0000 7f80967fc640 1 -- 172.21.15.156:0/546983466 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f80980512d0 con 0x7f80a01430c0 2024-03-25T18:14:53.793 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.792+0000 7f80967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:53.794 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.792+0000 7f80967fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:53.794 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:14:53.794 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:53.796 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.795+0000 7f80a8158640 10 librados: watch_flush enter 2024-03-25T18:14:53.796 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.795+0000 7f80a8158640 10 librados: watch_flush exit 2024-03-25T18:14:53.796 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.795+0000 7f80a8158640 20 client.4518.objecter shutdown clearing up homeless session... 2024-03-25T18:14:53.796 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.795+0000 7f80a8158640 10 client.4518.objecter successfully canceled tick 2024-03-25T18:14:53.797 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.795+0000 7f80a8158640 1 -- 172.21.15.156:0/546983466 >> v1:172.21.15.194:6800/2946427713 conn(0x7f808406ddf0 legacy=0x7f80840702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:53.797 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.795+0000 7f80a8158640 10 monclient: shutdown 2024-03-25T18:14:53.797 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.795+0000 7f80a8158640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:53.797 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.795+0000 7f80a8158640 1 -- 172.21.15.156:0/546983466 >> v1:172.21.15.156:6789/0 conn(0x7f80a01430c0 legacy=0x7f80a013ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:53.797 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.795+0000 7f80a8158640 1 -- 172.21.15.156:0/546983466 shutdown_connections 2024-03-25T18:14:53.797 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.795+0000 7f80a8158640 1 -- 172.21.15.156:0/546983466 >> 172.21.15.156:0/546983466 conn(0x7f80a00b7ea0 msgr2=0x7f80a0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:53.797 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.795+0000 7f80a8158640 1 -- 172.21.15.156:0/546983466 shutdown_connections 2024-03-25T18:14:53.797 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.795+0000 7f80a8158640 1 -- 172.21.15.156:0/546983466 wait complete. 2024-03-25T18:14:53.797 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:14:53.795+0000 7f80a8158640 1 librados: shutdown 2024-03-25T18:14:53.808 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:00.615 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:00.731 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.730+0000 7f681e74d640 10 monclient: get_monmap_and_config 2024-03-25T18:15:00.731 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.730+0000 7f681e74d640 10 monclient: build_initial_monmap 2024-03-25T18:15:00.731 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.730+0000 7f681e74d640 10 monclient: monmap: 2024-03-25T18:15:00.731 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:15:00.731 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:00.731 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:15:00.731082+0000 2024-03-25T18:15:00.731 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:15:00.731082+0000 2024-03-25T18:15:00.731 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:00.732 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:00.732 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:15:00.732 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:15:00.732 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:15:00.732 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:00.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.730+0000 7f681e74d640 1 Processor -- start 2024-03-25T18:15:00.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.731+0000 7f681e74d640 1 -- start start 2024-03-25T18:15:00.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.731+0000 7f681e74d640 10 monclient: init 2024-03-25T18:15:00.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.731+0000 7f681e74d640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:00.733 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.731+0000 7f681e74d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:15:00.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.731+0000 7f681e74d640 10 monclient(hunting): picked mon.noname-b con 0x7f68180a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:15:00.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.731+0000 7f681e74d640 10 monclient(hunting): picked mon.noname-c con 0x7f681813f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:15:00.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.731+0000 7f681e74d640 10 monclient(hunting): picked mon.noname-a con 0x7f68181430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:15:00.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.731+0000 7f681e74d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6818142520 con 0x7f68181430c0 2024-03-25T18:15:00.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.731+0000 7f681e74d640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68180b28a0 con 0x7f681813f710 2024-03-25T18:15:00.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.732+0000 7f681e74d640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68180b2a30 con 0x7f68180a98f0 2024-03-25T18:15:00.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.732+0000 7f681e74d640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:00.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.732+0000 7f681e74d640 10 monclient(hunting): authenticate will time out at 7382.042809s 2024-03-25T18:15:00.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.742+0000 7f68177fe640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f681813f710 0x7f68181429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:59054/0 (socket says 172.21.15.156:59054) 2024-03-25T18:15:00.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.742+0000 7f68177fe640 1 -- 172.21.15.156:0/3108863071 learned_addr learned my addr 172.21.15.156:0/3108863071 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:15:00.749 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.748+0000 7f6816ffd640 1 -- 172.21.15.156:0/3108863071 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 526516708 0 0) 0x7f68180b28a0 con 0x7f681813f710 2024-03-25T18:15:00.749 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.748+0000 7f6816ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:00.749 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.748+0000 7f6816ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:00.749 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.748+0000 7f6816ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:00.750 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.748+0000 7f6816ffd640 10 monclient(hunting): my global_id is 4528 2024-03-25T18:15:00.750 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.748+0000 7f6816ffd640 1 -- 172.21.15.156:0/3108863071 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6800003520 con 0x7f681813f710 2024-03-25T18:15:00.756 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.755+0000 7f6816ffd640 1 -- 172.21.15.156:0/3108863071 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2509326786 0 0) 0x7f68180b2a30 con 0x7f68180a98f0 2024-03-25T18:15:00.756 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.755+0000 7f6816ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:00.756 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.755+0000 7f6816ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:00.756 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.755+0000 7f6816ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:00.756 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.755+0000 7f6816ffd640 10 monclient(hunting): my global_id is 4532 2024-03-25T18:15:00.756 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.755+0000 7f6816ffd640 1 -- 172.21.15.156:0/3108863071 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68180b28a0 con 0x7f68180a98f0 2024-03-25T18:15:00.764 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.762+0000 7f6816ffd640 1 -- 172.21.15.156:0/3108863071 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1072329329 0 0) 0x7f6800003520 con 0x7f681813f710 2024-03-25T18:15:00.764 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.762+0000 7f6816ffd640 1 -- 172.21.15.156:0/3108863071 >> v1:172.21.15.156:6790/0 conn(0x7f68180a98f0 legacy=0x7f68180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:00.764 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.762+0000 7f6816ffd640 1 -- 172.21.15.156:0/3108863071 >> v1:172.21.15.156:6789/0 conn(0x7f68181430c0 legacy=0x7f681814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:15:00.765 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.762+0000 7f6816ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:00.765 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.762+0000 7f6816ffd640 1 monclient: found mon.noname-c 2024-03-25T18:15:00.765 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.762+0000 7f6816ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:00.765 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.762+0000 7f6816ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:15:00.765 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.762+0000 7f6816ffd640 1 -- 172.21.15.156:0/3108863071 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68180b44c0 con 0x7f681813f710 2024-03-25T18:15:00.765 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.762+0000 7f6816ffd640 10 monclient: _finish_auth 0 2024-03-25T18:15:00.765 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.762+0000 7f6816ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:00.765 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.762+0000 7f6816ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:00.765 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.762+0000 7f6816ffd640 1 -- 172.21.15.156:0/3108863071 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f680c001690 con 0x7f681813f710 2024-03-25T18:15:00.765 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.763+0000 7f6816ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:00.765 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.763+0000 7f6816ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:00.765 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.763+0000 7f6816ffd640 10 monclient: dump: 2024-03-25T18:15:00.765 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:00.766 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:00.766 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:00.766 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:00.766 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:00.766 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:00.766 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:00.766 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:00.766 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:00.766 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:00.766 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.763+0000 7f681e74d640 5 monclient: authenticate success, global_id 4528 2024-03-25T18:15:00.766 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.763+0000 7f681e74d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:00.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.776+0000 7f6816ffd640 1 -- 172.21.15.156:0/3108863071 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f680c0017f0 con 0x7f681813f710 2024-03-25T18:15:00.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.776+0000 7f6816ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:00.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.776+0000 7f6816ffd640 1 -- 172.21.15.156:0/3108863071 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f680c001c00 con 0x7f681813f710 2024-03-25T18:15:00.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.776+0000 7f681e74d640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:00.778 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.776+0000 7f681e74d640 4 set_mon_vals no callback set 2024-03-25T18:15:00.778 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.776+0000 7f6816ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:00.778 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.776+0000 7f6816ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:00.778 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.776+0000 7f6816ffd640 10 monclient: dump: 2024-03-25T18:15:00.778 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:00.778 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:00.778 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:00.778 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:00.778 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:00.778 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:00.778 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:00.778 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:00.778 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:00.778 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:00.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.777+0000 7f681e74d640 10 monclient: shutdown 2024-03-25T18:15:00.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.777+0000 7f681e74d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:00.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.777+0000 7f681e74d640 1 -- 172.21.15.156:0/3108863071 >> v1:172.21.15.194:6789/0 conn(0x7f681813f710 legacy=0x7f68181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:00.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.777+0000 7f681e74d640 1 -- 172.21.15.156:0/3108863071 shutdown_connections 2024-03-25T18:15:00.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.777+0000 7f681e74d640 1 -- 172.21.15.156:0/3108863071 >> 172.21.15.156:0/3108863071 conn(0x7f68180b7ea0 msgr2=0x7f68180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:00.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.777+0000 7f681e74d640 1 -- 172.21.15.156:0/3108863071 shutdown_connections 2024-03-25T18:15:00.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.777+0000 7f681e74d640 1 -- 172.21.15.156:0/3108863071 wait complete. 2024-03-25T18:15:00.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.778+0000 7f681e74d640 10 monclient: build_initial_monmap 2024-03-25T18:15:00.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.778+0000 7f681e74d640 10 monclient: monmap: 2024-03-25T18:15:00.782 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:15:00.782 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:00.782 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:15:00.782 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:15:00.782 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:00.782 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:15:00.782 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:15:00.782 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:15:00.782 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:15:00.782 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:00.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.778+0000 7f681e74d640 1 librados: starting msgr at 2024-03-25T18:15:00.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.778+0000 7f681e74d640 1 librados: starting objecter 2024-03-25T18:15:00.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.779+0000 7f681e74d640 1 Processor -- start 2024-03-25T18:15:00.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.779+0000 7f681e74d640 1 -- start start 2024-03-25T18:15:00.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.779+0000 7f681e74d640 1 librados: setting wanted keys 2024-03-25T18:15:00.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.779+0000 7f681e74d640 1 librados: calling monclient init 2024-03-25T18:15:00.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.779+0000 7f681e74d640 10 monclient: init 2024-03-25T18:15:00.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.779+0000 7f681e74d640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:00.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.779+0000 7f681e74d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:00.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.779+0000 7f681e74d640 10 monclient(hunting): picked mon.noname-a con 0x7f68180a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:15:00.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.779+0000 7f681e74d640 10 monclient(hunting): picked mon.noname-b con 0x7f681813f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:15:00.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.780+0000 7f681e74d640 10 monclient(hunting): picked mon.noname-c con 0x7f68181430c0 addr v1:172.21.15.156:6790/0 2024-03-25T18:15:00.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.780+0000 7f681e74d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f681813b2b0 con 0x7f68180a98f0 2024-03-25T18:15:00.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.780+0000 7f681e74d640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f681813b500 con 0x7f681813f710 2024-03-25T18:15:00.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.780+0000 7f681e74d640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68180b6f40 con 0x7f68181430c0 2024-03-25T18:15:00.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.780+0000 7f681e74d640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:00.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.780+0000 7f681e74d640 10 monclient(hunting): authenticate will time out at 7382.090932s 2024-03-25T18:15:00.792 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.791+0000 7f681ccea640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f68181430c0 0x7f68181e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:46376/0 (socket says 172.21.15.156:46376) 2024-03-25T18:15:00.792 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.791+0000 7f681ccea640 1 -- 172.21.15.156:0/191374360 learned_addr learned my addr 172.21.15.156:0/191374360 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:15:00.792 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.791+0000 7f6814ff9640 10 client.?.objecter ms_handle_connect 0x7f68181430c0 2024-03-25T18:15:00.792 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.791+0000 7f6814ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:00.793 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.792+0000 7f6814ff9640 10 client.?.objecter ms_handle_connect 0x7f681813f710 2024-03-25T18:15:00.793 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.792+0000 7f6814ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:00.793 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.792+0000 7f6814ff9640 10 client.?.objecter ms_handle_connect 0x7f68180a98f0 2024-03-25T18:15:00.793 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.792+0000 7f6814ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:00.806 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.805+0000 7f6814ff9640 1 -- 172.21.15.156:0/191374360 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580463808 0 0) 0x7f681813b2b0 con 0x7f68180a98f0 2024-03-25T18:15:00.806 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.805+0000 7f6814ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:00.806 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.805+0000 7f6814ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:00.806 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.805+0000 7f6814ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:00.806 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.805+0000 7f6814ff9640 10 monclient(hunting): my global_id is 4521 2024-03-25T18:15:00.806 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.805+0000 7f6814ff9640 1 -- 172.21.15.156:0/191374360 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6804004eb0 con 0x7f68180a98f0 2024-03-25T18:15:00.807 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.805+0000 7f6814ff9640 1 -- 172.21.15.156:0/191374360 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3158111411 0 0) 0x7f681813b500 con 0x7f681813f710 2024-03-25T18:15:00.807 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.805+0000 7f6814ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:00.807 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.805+0000 7f6814ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:00.807 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.805+0000 7f6814ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:00.807 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.805+0000 7f6814ff9640 10 monclient(hunting): my global_id is 4531 2024-03-25T18:15:00.807 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.805+0000 7f6814ff9640 1 -- 172.21.15.156:0/191374360 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f681813b2b0 con 0x7f681813f710 2024-03-25T18:15:00.815 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.814+0000 7f6814ff9640 1 -- 172.21.15.156:0/191374360 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4090278130 0 0) 0x7f68180b6f40 con 0x7f68181430c0 2024-03-25T18:15:00.815 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.814+0000 7f6814ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:00.815 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.814+0000 7f6814ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:00.815 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.814+0000 7f6814ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:00.815 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.814+0000 7f6814ff9640 10 monclient(hunting): my global_id is 4535 2024-03-25T18:15:00.815 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.814+0000 7f6814ff9640 1 -- 172.21.15.156:0/191374360 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f681813b500 con 0x7f68181430c0 2024-03-25T18:15:00.829 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.828+0000 7f6814ff9640 1 -- 172.21.15.156:0/191374360 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1668946762 0 0) 0x7f681813b500 con 0x7f68181430c0 2024-03-25T18:15:00.829 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.828+0000 7f6814ff9640 1 -- 172.21.15.156:0/191374360 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68180b6f40 con 0x7f68181430c0 2024-03-25T18:15:00.829 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.828+0000 7f6814ff9640 1 -- 172.21.15.156:0/191374360 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f6810001a10 con 0x7f68181430c0 2024-03-25T18:15:00.829 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.828+0000 7f6814ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:00.829 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.828+0000 7f6814ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:00.829 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.828+0000 7f6814ff9640 10 monclient(hunting): dump: 2024-03-25T18:15:00.829 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:00.829 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:00.830 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:00.830 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:00.830 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:00.830 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:00.830 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:00.830 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:00.830 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:00.830 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:00.830 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.828+0000 7f6814ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:00.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.830+0000 7f6814ff9640 1 -- 172.21.15.156:0/191374360 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 314827651 0 0) 0x7f681813b2b0 con 0x7f681813f710 2024-03-25T18:15:00.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.830+0000 7f6814ff9640 1 -- 172.21.15.156:0/191374360 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f681813b500 con 0x7f681813f710 2024-03-25T18:15:00.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.830+0000 7f6814ff9640 1 -- 172.21.15.156:0/191374360 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f680c001b00 con 0x7f681813f710 2024-03-25T18:15:00.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.830+0000 7f6814ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:00.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.830+0000 7f6814ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:00.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.830+0000 7f6814ff9640 10 monclient(hunting): dump: 2024-03-25T18:15:00.832 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:00.832 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:00.832 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:00.832 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:00.833 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:00.833 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:00.833 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:00.833 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:00.833 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:00.833 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:00.843 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.840+0000 7f6814ff9640 1 -- 172.21.15.156:0/191374360 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 859370819 0 0) 0x7f6804004eb0 con 0x7f68180a98f0 2024-03-25T18:15:00.843 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.840+0000 7f6814ff9640 1 -- 172.21.15.156:0/191374360 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f681813b2b0 con 0x7f68180a98f0 2024-03-25T18:15:00.843 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.840+0000 7f6814ff9640 1 -- 172.21.15.156:0/191374360 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f6808001990 con 0x7f68180a98f0 2024-03-25T18:15:00.843 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.840+0000 7f6814ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:00.843 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.840+0000 7f6814ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:00.843 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.840+0000 7f6814ff9640 10 monclient(hunting): dump: 2024-03-25T18:15:00.843 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:00.843 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:00.843 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:00.843 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:00.843 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:00.843 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:00.843 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:00.844 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:00.844 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:00.844 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:00.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.844+0000 7f6814ff9640 1 -- 172.21.15.156:0/191374360 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3391100751 0 0) 0x7f681813b500 con 0x7f681813f710 2024-03-25T18:15:00.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.844+0000 7f6814ff9640 1 -- 172.21.15.156:0/191374360 >> v1:172.21.15.156:6790/0 conn(0x7f68181430c0 legacy=0x7f68181e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:00.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.844+0000 7f6814ff9640 1 -- 172.21.15.156:0/191374360 >> v1:172.21.15.156:6789/0 conn(0x7f68180a98f0 legacy=0x7f6818139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:00.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.844+0000 7f6814ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:15:00.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.844+0000 7f6814ff9640 1 monclient: found mon.b 2024-03-25T18:15:00.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.844+0000 7f6814ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:00.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.844+0000 7f6814ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:15:00.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.844+0000 7f6814ff9640 1 -- 172.21.15.156:0/191374360 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681813a430 con 0x7f681813f710 2024-03-25T18:15:00.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.844+0000 7f6814ff9640 10 monclient: _finish_auth 0 2024-03-25T18:15:00.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.844+0000 7f6814ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:15:00.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.844+0000 7f6814ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:00.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.844+0000 7f681e74d640 5 monclient: authenticate success, global_id 4531 2024-03-25T18:15:00.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.844+0000 7f681e74d640 10 monclient: _renew_subs 2024-03-25T18:15:00.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.844+0000 7f681e74d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:15:00.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.844+0000 7f681e74d640 1 -- 172.21.15.156:0/191374360 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f681812dd10 con 0x7f681813f710 2024-03-25T18:15:00.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.844+0000 7f681e74d640 10 client.4531.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:00.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.844+0000 7f681e74d640 10 monclient: _renew_subs 2024-03-25T18:15:00.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.844+0000 7f681e74d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:15:00.848 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.844+0000 7f681e74d640 1 -- 172.21.15.156:0/191374360 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68180b7b40 con 0x7f681813f710 2024-03-25T18:15:00.848 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.844+0000 7f681e74d640 1 librados: init done 2024-03-25T18:15:00.848 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.845+0000 7f681e74d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:00.848 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.845+0000 7f681e74d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:00.848 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.845+0000 7f681e74d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:15:00.848 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:00.845+0000 7f681e74d640 1 -- 172.21.15.156:0/191374360 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68180b7130 con 0x7f681813f710 2024-03-25T18:15:01.139 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.137+0000 7f6814ff9640 1 -- 172.21.15.156:0/191374360 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f680c001ca0 con 0x7f681813f710 2024-03-25T18:15:01.139 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.137+0000 7f6814ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:01.139 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.137+0000 7f6814ff9640 1 -- 172.21.15.156:0/191374360 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f680c004c30 con 0x7f681813f710 2024-03-25T18:15:01.139 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.137+0000 7f6814ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:01.139 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.137+0000 7f6814ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:01.139 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.137+0000 7f6814ff9640 10 monclient: dump: 2024-03-25T18:15:01.140 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:01.140 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:01.140 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:01.140 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:01.140 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:01.140 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:01.140 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:01.140 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:01.140 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:01.140 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:01.140 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.137+0000 7f68167fc640 4 set_mon_vals no callback set 2024-03-25T18:15:01.141 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.137+0000 7f6814ff9640 1 -- 172.21.15.156:0/191374360 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f680c01ac00 con 0x7f681813f710 2024-03-25T18:15:01.141 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.138+0000 7f6817fff640 10 monclient: get_auth_request con 0x7f680406e110 auth_method 0 2024-03-25T18:15:01.141 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.139+0000 7f6814ff9640 10 client.4531.objecter ms_handle_connect 0x7f680406e110 2024-03-25T18:15:01.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.142+0000 7f6814ff9640 1 -- 172.21.15.156:0/191374360 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f680c054800 con 0x7f681813f710 2024-03-25T18:15:01.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.142+0000 7f6814ff9640 10 client.4531.objecter ms_dispatch 0x7f6818000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:01.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.142+0000 7f6814ff9640 3 client.4531.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:01.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.142+0000 7f6814ff9640 3 client.4531.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:01.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.142+0000 7f6814ff9640 20 client.4531.objecter dump_active .. 0 homeless 2024-03-25T18:15:01.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.142+0000 7f6814ff9640 1 -- 172.21.15.156:0/191374360 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f680c050310 con 0x7f681813f710 2024-03-25T18:15:01.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.142+0000 7f6814ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:01.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.142+0000 7f6814ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:01.393 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.391+0000 7f681e74d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:01.393 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.391+0000 7f681e74d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:01.393 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.391+0000 7f681e74d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:15:01.393 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.391+0000 7f681e74d640 1 -- 172.21.15.156:0/191374360 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68180627b0 con 0x7f681813f710 2024-03-25T18:15:01.425 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.423+0000 7f6814ff9640 1 -- 172.21.15.156:0/191374360 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f680c052240 con 0x7f681813f710 2024-03-25T18:15:01.425 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.423+0000 7f6814ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:01.425 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.423+0000 7f6814ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:01.425 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:15:01.426 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:01.428 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.425+0000 7f681e74d640 10 librados: watch_flush enter 2024-03-25T18:15:01.428 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.425+0000 7f681e74d640 10 librados: watch_flush exit 2024-03-25T18:15:01.428 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.425+0000 7f681e74d640 20 client.4531.objecter shutdown clearing up homeless session... 2024-03-25T18:15:01.428 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.425+0000 7f681e74d640 10 client.4531.objecter successfully canceled tick 2024-03-25T18:15:01.428 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.426+0000 7f681e74d640 1 -- 172.21.15.156:0/191374360 >> v1:172.21.15.194:6800/2946427713 conn(0x7f680406e110 legacy=0x7f68040705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:01.428 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.426+0000 7f681e74d640 10 monclient: shutdown 2024-03-25T18:15:01.428 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.426+0000 7f681e74d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:01.429 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.426+0000 7f681e74d640 1 -- 172.21.15.156:0/191374360 >> v1:172.21.15.194:6789/0 conn(0x7f681813f710 legacy=0x7f68181df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:01.429 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.426+0000 7f681e74d640 1 -- 172.21.15.156:0/191374360 shutdown_connections 2024-03-25T18:15:01.429 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.426+0000 7f681e74d640 1 -- 172.21.15.156:0/191374360 >> 172.21.15.156:0/191374360 conn(0x7f68180b7ea0 msgr2=0x7f6818074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:01.429 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.426+0000 7f681e74d640 1 -- 172.21.15.156:0/191374360 shutdown_connections 2024-03-25T18:15:01.429 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.426+0000 7f681e74d640 1 -- 172.21.15.156:0/191374360 wait complete. 2024-03-25T18:15:01.429 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:01.426+0000 7f681e74d640 1 librados: shutdown 2024-03-25T18:15:01.440 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:08.247 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:08.346 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.345+0000 7feeaaed9640 10 monclient: get_monmap_and_config 2024-03-25T18:15:08.346 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.345+0000 7feeaaed9640 10 monclient: build_initial_monmap 2024-03-25T18:15:08.346 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.345+0000 7feeaaed9640 10 monclient: monmap: 2024-03-25T18:15:08.346 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:15:08.346 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:08.346 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:15:08.346163+0000 2024-03-25T18:15:08.346 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:15:08.346163+0000 2024-03-25T18:15:08.346 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:08.347 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:08.347 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:15:08.347 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:15:08.347 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:15:08.347 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:08.348 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.345+0000 7feeaaed9640 1 Processor -- start 2024-03-25T18:15:08.348 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.346+0000 7feeaaed9640 1 -- start start 2024-03-25T18:15:08.348 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.346+0000 7feeaaed9640 10 monclient: init 2024-03-25T18:15:08.348 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.346+0000 7feeaaed9640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:08.348 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.346+0000 7feeaaed9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:08.348 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.346+0000 7feeaaed9640 10 monclient(hunting): picked mon.noname-b con 0x7feea40b4d60 addr v1:172.21.15.156:6790/0 2024-03-25T18:15:08.348 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.346+0000 7feeaaed9640 10 monclient(hunting): picked mon.noname-a con 0x7feea40b1730 addr v1:172.21.15.156:6789/0 2024-03-25T18:15:08.348 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.346+0000 7feeaaed9640 10 monclient(hunting): picked mon.noname-c con 0x7feea414be90 addr v1:172.21.15.194:6789/0 2024-03-25T18:15:08.348 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.346+0000 7feeaaed9640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea40af940 con 0x7feea40b1730 2024-03-25T18:15:08.349 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.346+0000 7feeaaed9640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea40b0a70 con 0x7feea414be90 2024-03-25T18:15:08.349 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.346+0000 7feeaaed9640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea40b0d50 con 0x7feea40b4d60 2024-03-25T18:15:08.349 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.346+0000 7feeaaed9640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:08.349 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.346+0000 7feeaaed9640 10 monclient(hunting): authenticate will time out at 7389.657474s 2024-03-25T18:15:08.363 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.362+0000 7feea3fff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7feea40b1730 0x7feea40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:51034/0 (socket says 172.21.15.156:51034) 2024-03-25T18:15:08.363 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.362+0000 7feea3fff640 1 -- 172.21.15.156:0/3395659927 learned_addr learned my addr 172.21.15.156:0/3395659927 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:15:08.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.380+0000 7feea37fe640 1 -- 172.21.15.156:0/3395659927 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766677558 0 0) 0x7feea40af940 con 0x7feea40b1730 2024-03-25T18:15:08.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.380+0000 7feea37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:08.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.380+0000 7feea37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:08.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.380+0000 7feea37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:08.382 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.380+0000 7feea37fe640 10 monclient(hunting): my global_id is 4524 2024-03-25T18:15:08.382 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.380+0000 7feea37fe640 1 -- 172.21.15.156:0/3395659927 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee94003520 con 0x7feea40b1730 2024-03-25T18:15:08.390 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.389+0000 7feea37fe640 1 -- 172.21.15.156:0/3395659927 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1764340968 0 0) 0x7fee94003520 con 0x7feea40b1730 2024-03-25T18:15:08.390 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.389+0000 7feea37fe640 1 -- 172.21.15.156:0/3395659927 >> v1:172.21.15.156:6790/0 conn(0x7feea40b4d60 legacy=0x7feea413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:15:08.390 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.389+0000 7feea37fe640 1 -- 172.21.15.156:0/3395659927 >> v1:172.21.15.194:6789/0 conn(0x7feea414be90 legacy=0x7feea414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:15:08.390 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.389+0000 7feea37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:15:08.390 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.389+0000 7feea37fe640 1 monclient: found mon.noname-a 2024-03-25T18:15:08.390 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.389+0000 7feea37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:08.390 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.389+0000 7feea37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:15:08.390 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.389+0000 7feea37fe640 1 -- 172.21.15.156:0/3395659927 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea40b1030 con 0x7feea40b1730 2024-03-25T18:15:08.390 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.389+0000 7feea37fe640 10 monclient: _finish_auth 0 2024-03-25T18:15:08.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.389+0000 7feea37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:15:08.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.389+0000 7feea37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:08.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.389+0000 7feea37fe640 1 -- 172.21.15.156:0/3395659927 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fee90001690 con 0x7feea40b1730 2024-03-25T18:15:08.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.389+0000 7feeaaed9640 5 monclient: authenticate success, global_id 4524 2024-03-25T18:15:08.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.389+0000 7feeaaed9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:08.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.389+0000 7feea37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:08.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.389+0000 7feea37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:08.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.389+0000 7feea37fe640 10 monclient: dump: 2024-03-25T18:15:08.391 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:08.391 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:08.391 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:08.391 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:08.391 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:08.391 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:08.391 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:08.392 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:08.392 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:08.392 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:08.392 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.389+0000 7feeaaed9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:08.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.408+0000 7feea37fe640 1 -- 172.21.15.156:0/3395659927 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee90001860 con 0x7feea40b1730 2024-03-25T18:15:08.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.408+0000 7feea37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:08.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.408+0000 7feea37fe640 1 -- 172.21.15.156:0/3395659927 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fee90001cb0 con 0x7feea40b1730 2024-03-25T18:15:08.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.408+0000 7feea37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:08.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.408+0000 7feea37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:08.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.408+0000 7feea37fe640 10 monclient: dump: 2024-03-25T18:15:08.409 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:08.409 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:08.410 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:08.410 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:08.410 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:08.410 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:08.410 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:08.410 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:08.410 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:08.410 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:08.410 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.408+0000 7feeaaed9640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:08.410 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.408+0000 7feeaaed9640 4 set_mon_vals no callback set 2024-03-25T18:15:08.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.409+0000 7feeaaed9640 10 monclient: shutdown 2024-03-25T18:15:08.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.409+0000 7feeaaed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:08.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.409+0000 7feeaaed9640 1 -- 172.21.15.156:0/3395659927 >> v1:172.21.15.156:6789/0 conn(0x7feea40b1730 legacy=0x7feea40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:08.411 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.409+0000 7feeaaed9640 1 -- 172.21.15.156:0/3395659927 shutdown_connections 2024-03-25T18:15:08.412 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.409+0000 7feeaaed9640 1 -- 172.21.15.156:0/3395659927 >> 172.21.15.156:0/3395659927 conn(0x7feea40617a0 msgr2=0x7feea4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:08.412 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.409+0000 7feeaaed9640 1 -- 172.21.15.156:0/3395659927 shutdown_connections 2024-03-25T18:15:08.412 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.409+0000 7feeaaed9640 1 -- 172.21.15.156:0/3395659927 wait complete. 2024-03-25T18:15:08.412 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.410+0000 7feeaaed9640 10 monclient: build_initial_monmap 2024-03-25T18:15:08.412 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.410+0000 7feeaaed9640 10 monclient: monmap: 2024-03-25T18:15:08.412 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:15:08.412 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:08.412 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:15:08.412 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:15:08.412 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:08.412 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:15:08.412 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:15:08.412 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:15:08.412 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:15:08.412 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:08.412 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.410+0000 7feeaaed9640 1 librados: starting msgr at 2024-03-25T18:15:08.413 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.410+0000 7feeaaed9640 1 librados: starting objecter 2024-03-25T18:15:08.413 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.410+0000 7feeaaed9640 1 Processor -- start 2024-03-25T18:15:08.413 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.410+0000 7feeaaed9640 1 -- start start 2024-03-25T18:15:08.413 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.410+0000 7feeaaed9640 1 librados: setting wanted keys 2024-03-25T18:15:08.413 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.410+0000 7feeaaed9640 1 librados: calling monclient init 2024-03-25T18:15:08.413 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.410+0000 7feeaaed9640 10 monclient: init 2024-03-25T18:15:08.414 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.411+0000 7feeaaed9640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:08.414 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.411+0000 7feeaaed9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:15:08.414 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.411+0000 7feeaaed9640 10 monclient(hunting): picked mon.noname-c con 0x7feea40b1730 addr v1:172.21.15.156:6790/0 2024-03-25T18:15:08.414 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.411+0000 7feeaaed9640 10 monclient(hunting): picked mon.noname-b con 0x7feea40b4d60 addr v1:172.21.15.194:6789/0 2024-03-25T18:15:08.414 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.411+0000 7feeaaed9640 10 monclient(hunting): picked mon.noname-a con 0x7feea414be90 addr v1:172.21.15.156:6789/0 2024-03-25T18:15:08.414 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.411+0000 7feeaaed9640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea40a9b80 con 0x7feea414be90 2024-03-25T18:15:08.414 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.411+0000 7feeaaed9640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea40aacd0 con 0x7feea40b4d60 2024-03-25T18:15:08.414 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.411+0000 7feeaaed9640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea41dafd0 con 0x7feea40b1730 2024-03-25T18:15:08.414 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.411+0000 7feeaaed9640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:08.414 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.411+0000 7feeaaed9640 10 monclient(hunting): authenticate will time out at 7389.722643s 2024-03-25T18:15:08.414 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.413+0000 7feea3fff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7feea40b4d60 0x7feea41db4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:51668/0 (socket says 172.21.15.156:51668) 2024-03-25T18:15:08.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.413+0000 7feea3fff640 1 -- 172.21.15.156:0/2656137294 learned_addr learned my addr 172.21.15.156:0/2656137294 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:15:08.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.414+0000 7feea17fa640 10 client.?.objecter ms_handle_connect 0x7feea40b4d60 2024-03-25T18:15:08.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.414+0000 7feea17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:08.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.431+0000 7feea17fa640 10 client.?.objecter ms_handle_connect 0x7feea40b1730 2024-03-25T18:15:08.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.431+0000 7feea17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:08.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.434+0000 7feea17fa640 10 client.?.objecter ms_handle_connect 0x7feea414be90 2024-03-25T18:15:08.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.434+0000 7feea17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:08.489 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.488+0000 7feea17fa640 1 -- 172.21.15.156:0/2656137294 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4166415256 0 0) 0x7feea40a9b80 con 0x7feea414be90 2024-03-25T18:15:08.489 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.488+0000 7feea17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:08.489 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.488+0000 7feea17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:08.489 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.488+0000 7feea17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:08.489 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.488+0000 7feea17fa640 10 monclient(hunting): my global_id is 4527 2024-03-25T18:15:08.489 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.488+0000 7feea17fa640 1 -- 172.21.15.156:0/2656137294 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee88004eb0 con 0x7feea414be90 2024-03-25T18:15:08.506 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.505+0000 7feea17fa640 1 -- 172.21.15.156:0/2656137294 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675582255 0 0) 0x7feea41dafd0 con 0x7feea40b1730 2024-03-25T18:15:08.506 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.505+0000 7feea17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:08.506 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.505+0000 7feea17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:08.506 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.505+0000 7feea17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:08.506 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.505+0000 7feea17fa640 10 monclient(hunting): my global_id is 4538 2024-03-25T18:15:08.506 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.505+0000 7feea17fa640 1 -- 172.21.15.156:0/2656137294 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feea40a9b80 con 0x7feea40b1730 2024-03-25T18:15:08.514 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.513+0000 7feea17fa640 1 -- 172.21.15.156:0/2656137294 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1222766505 0 0) 0x7fee88004eb0 con 0x7feea414be90 2024-03-25T18:15:08.514 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.513+0000 7feea17fa640 1 -- 172.21.15.156:0/2656137294 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feea41dafd0 con 0x7feea414be90 2024-03-25T18:15:08.514 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.513+0000 7feea17fa640 1 -- 172.21.15.156:0/2656137294 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fee9c001ad0 con 0x7feea414be90 2024-03-25T18:15:08.514 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.513+0000 7feea17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:08.514 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.513+0000 7feea17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:08.514 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.513+0000 7feea17fa640 10 monclient(hunting): dump: 2024-03-25T18:15:08.514 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:08.514 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:08.514 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:08.515 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:08.515 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:08.515 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:08.515 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:08.515 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:08.515 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:08.515 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:08.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.513+0000 7feea17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:08.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.525+0000 7feea17fa640 1 -- 172.21.15.156:0/2656137294 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3197707902 0 0) 0x7feea40a9b80 con 0x7feea40b1730 2024-03-25T18:15:08.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.525+0000 7feea17fa640 1 -- 172.21.15.156:0/2656137294 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee88004eb0 con 0x7feea40b1730 2024-03-25T18:15:08.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.525+0000 7feea17fa640 1 -- 172.21.15.156:0/2656137294 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fee98001810 con 0x7feea40b1730 2024-03-25T18:15:08.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.525+0000 7feea17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:08.527 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.525+0000 7feea17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:08.527 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.525+0000 7feea17fa640 10 monclient(hunting): dump: 2024-03-25T18:15:08.527 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:08.527 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:08.527 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:08.527 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:08.527 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:08.527 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:08.527 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:08.527 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:08.527 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:08.527 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:08.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.533+0000 7feea17fa640 1 -- 172.21.15.156:0/2656137294 <== mon.2 v1:172.21.15.156:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 890404183 0 0) 0x7fee88004eb0 con 0x7feea40b1730 2024-03-25T18:15:08.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.533+0000 7feea17fa640 1 -- 172.21.15.156:0/2656137294 >> v1:172.21.15.194:6789/0 conn(0x7feea40b4d60 legacy=0x7feea41db4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:08.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.533+0000 7feea17fa640 1 -- 172.21.15.156:0/2656137294 >> v1:172.21.15.156:6789/0 conn(0x7feea414be90 legacy=0x7feea41e2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:08.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.533+0000 7feea17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:15:08.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.533+0000 7feea17fa640 1 monclient: found mon.c 2024-03-25T18:15:08.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.533+0000 7feea17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:08.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.533+0000 7feea17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:15:08.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.533+0000 7feea17fa640 1 -- 172.21.15.156:0/2656137294 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea4138d60 con 0x7feea40b1730 2024-03-25T18:15:08.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.533+0000 7feea17fa640 10 monclient: _finish_auth 0 2024-03-25T18:15:08.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.533+0000 7feea17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:15:08.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.533+0000 7feea17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:08.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.533+0000 7feeaaed9640 5 monclient: authenticate success, global_id 4538 2024-03-25T18:15:08.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.533+0000 7feeaaed9640 10 monclient: _renew_subs 2024-03-25T18:15:08.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.533+0000 7feeaaed9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:15:08.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.533+0000 7feeaaed9640 1 -- 172.21.15.156:0/2656137294 --> v1:172.21.15.156:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea41dbbe0 con 0x7feea40b1730 2024-03-25T18:15:08.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.533+0000 7feeaaed9640 10 client.4538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:08.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.533+0000 7feeaaed9640 10 monclient: _renew_subs 2024-03-25T18:15:08.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.533+0000 7feeaaed9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:15:08.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.533+0000 7feeaaed9640 1 -- 172.21.15.156:0/2656137294 --> v1:172.21.15.156:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feea413b7c0 con 0x7feea40b1730 2024-03-25T18:15:08.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.533+0000 7feeaaed9640 1 librados: init done 2024-03-25T18:15:08.536 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.534+0000 7feeaaed9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:08.536 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.534+0000 7feeaaed9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:08.536 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.534+0000 7feeaaed9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:15:08.536 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.534+0000 7feeaaed9640 1 -- 172.21.15.156:0/2656137294 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea41401f0 con 0x7feea40b1730 2024-03-25T18:15:08.862 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.860+0000 7feea17fa640 1 -- 172.21.15.156:0/2656137294 <== mon.2 v1:172.21.15.156:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee98001d00 con 0x7feea40b1730 2024-03-25T18:15:08.862 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.860+0000 7feea17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:08.862 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.860+0000 7feea17fa640 1 -- 172.21.15.156:0/2656137294 <== mon.2 v1:172.21.15.156:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fee98003790 con 0x7feea40b1730 2024-03-25T18:15:08.862 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.860+0000 7feea17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:08.862 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.860+0000 7feea17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:08.862 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.860+0000 7feea17fa640 10 monclient: dump: 2024-03-25T18:15:08.862 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:08.862 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:08.862 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:08.862 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:08.862 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:08.862 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:08.863 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:08.863 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:08.863 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:08.863 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:08.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.860+0000 7feea2ffd640 4 set_mon_vals no callback set 2024-03-25T18:15:08.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.861+0000 7feea17fa640 1 -- 172.21.15.156:0/2656137294 <== mon.2 v1:172.21.15.156:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7fee98019760 con 0x7feea40b1730 2024-03-25T18:15:08.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.862+0000 7feea3fff640 10 monclient: get_auth_request con 0x7fee8806ded0 auth_method 0 2024-03-25T18:15:08.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.862+0000 7feea17fa640 10 client.4538.objecter ms_handle_connect 0x7fee8806ded0 2024-03-25T18:15:08.872 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.870+0000 7feea17fa640 1 -- 172.21.15.156:0/2656137294 <== mon.2 v1:172.21.15.156:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7fee98053510 con 0x7feea40b1730 2024-03-25T18:15:08.872 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.870+0000 7feea17fa640 10 client.4538.objecter ms_dispatch 0x7feea4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:08.872 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.870+0000 7feea17fa640 3 client.4538.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:08.872 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.870+0000 7feea17fa640 3 client.4538.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:08.872 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.870+0000 7feea17fa640 20 client.4538.objecter dump_active .. 0 homeless 2024-03-25T18:15:08.872 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.870+0000 7feea17fa640 1 -- 172.21.15.156:0/2656137294 <== mon.2 v1:172.21.15.156:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee9804ee50 con 0x7feea40b1730 2024-03-25T18:15:08.872 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.871+0000 7feea17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:08.872 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:08.871+0000 7feea17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:09.125 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:09.124+0000 7feeaaed9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:09.125 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:09.124+0000 7feeaaed9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:09.125 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:09.124+0000 7feeaaed9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:15:09.125 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:09.124+0000 7feeaaed9640 1 -- 172.21.15.156:0/2656137294 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feea41db160 con 0x7feea40b1730 2024-03-25T18:15:09.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:09.157+0000 7feea17fa640 1 -- 172.21.15.156:0/2656137294 <== mon.2 v1:172.21.15.156:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7fee98050f50 con 0x7feea40b1730 2024-03-25T18:15:09.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:09.157+0000 7feea17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:09.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:09.157+0000 7feea17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:09.159 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:15:09.159 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:09.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:09.160+0000 7feeaaed9640 10 librados: watch_flush enter 2024-03-25T18:15:09.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:09.160+0000 7feeaaed9640 10 librados: watch_flush exit 2024-03-25T18:15:09.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:09.160+0000 7feeaaed9640 20 client.4538.objecter shutdown clearing up homeless session... 2024-03-25T18:15:09.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:09.160+0000 7feeaaed9640 10 client.4538.objecter successfully canceled tick 2024-03-25T18:15:09.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:09.160+0000 7feeaaed9640 1 -- 172.21.15.156:0/2656137294 >> v1:172.21.15.194:6800/2946427713 conn(0x7fee8806ded0 legacy=0x7fee88070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:09.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:09.160+0000 7feeaaed9640 10 monclient: shutdown 2024-03-25T18:15:09.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:09.160+0000 7feeaaed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:09.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:09.160+0000 7feeaaed9640 1 -- 172.21.15.156:0/2656137294 >> v1:172.21.15.156:6790/0 conn(0x7feea40b1730 legacy=0x7feea40a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:09.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:09.160+0000 7feeaaed9640 1 -- 172.21.15.156:0/2656137294 shutdown_connections 2024-03-25T18:15:09.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:09.160+0000 7feeaaed9640 1 -- 172.21.15.156:0/2656137294 >> 172.21.15.156:0/2656137294 conn(0x7feea40617a0 msgr2=0x7feea4144f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:09.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:09.160+0000 7feeaaed9640 1 -- 172.21.15.156:0/2656137294 shutdown_connections 2024-03-25T18:15:09.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:09.160+0000 7feeaaed9640 1 -- 172.21.15.156:0/2656137294 wait complete. 2024-03-25T18:15:09.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:09.160+0000 7feeaaed9640 1 librados: shutdown 2024-03-25T18:15:09.173 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:15.980 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:16.100 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.098+0000 7fec64d58640 10 monclient: get_monmap_and_config 2024-03-25T18:15:16.100 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.098+0000 7fec64d58640 10 monclient: build_initial_monmap 2024-03-25T18:15:16.100 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.098+0000 7fec64d58640 10 monclient: monmap: 2024-03-25T18:15:16.100 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:15:16.100 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:16.100 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:15:16.099664+0000 2024-03-25T18:15:16.100 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:15:16.099664+0000 2024-03-25T18:15:16.100 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:16.100 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:16.100 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:15:16.100 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:15:16.101 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:15:16.101 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:16.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.099+0000 7fec64d58640 1 Processor -- start 2024-03-25T18:15:16.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.099+0000 7fec64d58640 1 -- start start 2024-03-25T18:15:16.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.099+0000 7fec64d58640 10 monclient: init 2024-03-25T18:15:16.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.100+0000 7fec64d58640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:16.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.100+0000 7fec64d58640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:16.103 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.100+0000 7fec64d58640 10 monclient(hunting): picked mon.noname-a con 0x7fec600a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:15:16.103 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.100+0000 7fec64d58640 10 monclient(hunting): picked mon.noname-c con 0x7fec6013f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:15:16.103 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.100+0000 7fec64d58640 10 monclient(hunting): picked mon.noname-b con 0x7fec601430c0 addr v1:172.21.15.156:6790/0 2024-03-25T18:15:16.103 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.100+0000 7fec64d58640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec60142520 con 0x7fec600a98f0 2024-03-25T18:15:16.103 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.100+0000 7fec64d58640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec600b28a0 con 0x7fec6013f710 2024-03-25T18:15:16.103 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.100+0000 7fec64d58640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec600b2a30 con 0x7fec601430c0 2024-03-25T18:15:16.103 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.100+0000 7fec64d58640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:16.103 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.100+0000 7fec64d58640 10 monclient(hunting): authenticate will time out at 7397.411031s 2024-03-25T18:15:16.109 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.108+0000 7fec57fff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fec6013f710 0x7fec601429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:35586/0 (socket says 172.21.15.156:35586) 2024-03-25T18:15:16.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.108+0000 7fec57fff640 1 -- 172.21.15.156:0/4051887609 learned_addr learned my addr 172.21.15.156:0/4051887609 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:15:16.125 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.124+0000 7fec5dd9b640 1 -- 172.21.15.156:0/4051887609 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1737290121 0 0) 0x7fec60142520 con 0x7fec600a98f0 2024-03-25T18:15:16.125 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.124+0000 7fec5dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:15:16.125 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.124+0000 7fec5dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:16.126 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.124+0000 7fec5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:16.126 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.124+0000 7fec5dd9b640 10 monclient(hunting): my global_id is 4530 2024-03-25T18:15:16.126 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.124+0000 7fec5dd9b640 1 -- 172.21.15.156:0/4051887609 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec4c003520 con 0x7fec600a98f0 2024-03-25T18:15:16.126 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.125+0000 7fec5dd9b640 1 -- 172.21.15.156:0/4051887609 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719871655 0 0) 0x7fec600b2a30 con 0x7fec601430c0 2024-03-25T18:15:16.126 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.125+0000 7fec5dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:15:16.126 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.125+0000 7fec5dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:16.126 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.125+0000 7fec5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:16.127 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.125+0000 7fec5dd9b640 10 monclient(hunting): my global_id is 4541 2024-03-25T18:15:16.127 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.125+0000 7fec5dd9b640 1 -- 172.21.15.156:0/4051887609 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec60142520 con 0x7fec601430c0 2024-03-25T18:15:16.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.133+0000 7fec5dd9b640 1 -- 172.21.15.156:0/4051887609 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3071176496 0 0) 0x7fec4c003520 con 0x7fec600a98f0 2024-03-25T18:15:16.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.133+0000 7fec5dd9b640 1 -- 172.21.15.156:0/4051887609 >> v1:172.21.15.156:6790/0 conn(0x7fec601430c0 legacy=0x7fec6014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:16.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.133+0000 7fec5dd9b640 1 -- 172.21.15.156:0/4051887609 >> v1:172.21.15.194:6789/0 conn(0x7fec6013f710 legacy=0x7fec601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:16.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.133+0000 7fec5dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:15:16.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.133+0000 7fec5dd9b640 1 monclient: found mon.noname-a 2024-03-25T18:15:16.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.133+0000 7fec5dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:16.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.133+0000 7fec5dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:15:16.135 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.133+0000 7fec5dd9b640 1 -- 172.21.15.156:0/4051887609 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec600b44c0 con 0x7fec600a98f0 2024-03-25T18:15:16.135 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.133+0000 7fec5dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:15:16.135 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.133+0000 7fec5dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:15:16.135 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.133+0000 7fec5dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:16.135 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.133+0000 7fec5dd9b640 1 -- 172.21.15.156:0/4051887609 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fec50001690 con 0x7fec600a98f0 2024-03-25T18:15:16.135 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.133+0000 7fec64d58640 5 monclient: authenticate success, global_id 4530 2024-03-25T18:15:16.135 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.133+0000 7fec5dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:16.135 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.133+0000 7fec5dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:16.135 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.133+0000 7fec5dd9b640 10 monclient: dump: 2024-03-25T18:15:16.135 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:16.135 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:16.135 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:16.135 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:16.136 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:16.136 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:16.136 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:16.136 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:16.136 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:16.136 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:16.136 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.133+0000 7fec64d58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:16.141 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.140+0000 7fec5dd9b640 1 -- 172.21.15.156:0/4051887609 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec500017f0 con 0x7fec600a98f0 2024-03-25T18:15:16.141 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.140+0000 7fec5dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:16.141 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.140+0000 7fec5dd9b640 1 -- 172.21.15.156:0/4051887609 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fec50001c00 con 0x7fec600a98f0 2024-03-25T18:15:16.142 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.140+0000 7fec64d58640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:16.142 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.140+0000 7fec64d58640 4 set_mon_vals no callback set 2024-03-25T18:15:16.142 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.140+0000 7fec5dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:16.142 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.140+0000 7fec5dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:16.142 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.140+0000 7fec5dd9b640 10 monclient: dump: 2024-03-25T18:15:16.142 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:16.142 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:16.142 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:16.142 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:16.142 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:16.142 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:16.142 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:16.142 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:16.143 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:16.143 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:16.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.141+0000 7fec64d58640 10 monclient: shutdown 2024-03-25T18:15:16.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.141+0000 7fec64d58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:16.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.141+0000 7fec64d58640 1 -- 172.21.15.156:0/4051887609 >> v1:172.21.15.156:6789/0 conn(0x7fec600a98f0 legacy=0x7fec600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:16.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.141+0000 7fec64d58640 1 -- 172.21.15.156:0/4051887609 shutdown_connections 2024-03-25T18:15:16.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.141+0000 7fec64d58640 1 -- 172.21.15.156:0/4051887609 >> 172.21.15.156:0/4051887609 conn(0x7fec600b7ea0 msgr2=0x7fec600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:16.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.141+0000 7fec64d58640 1 -- 172.21.15.156:0/4051887609 shutdown_connections 2024-03-25T18:15:16.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.141+0000 7fec64d58640 1 -- 172.21.15.156:0/4051887609 wait complete. 2024-03-25T18:15:16.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.142+0000 7fec64d58640 10 monclient: build_initial_monmap 2024-03-25T18:15:16.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.142+0000 7fec64d58640 10 monclient: monmap: 2024-03-25T18:15:16.145 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:15:16.145 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:16.145 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:15:16.145 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:15:16.145 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:16.145 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:15:16.145 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:15:16.145 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:15:16.145 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:15:16.145 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:16.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.143+0000 7fec64d58640 1 librados: starting msgr at 2024-03-25T18:15:16.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.143+0000 7fec64d58640 1 librados: starting objecter 2024-03-25T18:15:16.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.143+0000 7fec64d58640 1 Processor -- start 2024-03-25T18:15:16.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.143+0000 7fec64d58640 1 -- start start 2024-03-25T18:15:16.146 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.143+0000 7fec64d58640 1 librados: setting wanted keys 2024-03-25T18:15:16.146 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.143+0000 7fec64d58640 1 librados: calling monclient init 2024-03-25T18:15:16.146 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.143+0000 7fec64d58640 10 monclient: init 2024-03-25T18:15:16.146 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.144+0000 7fec64d58640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:16.146 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.144+0000 7fec64d58640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:15:16.147 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.144+0000 7fec64d58640 10 monclient(hunting): picked mon.noname-b con 0x7fec600a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:15:16.147 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.144+0000 7fec64d58640 10 monclient(hunting): picked mon.noname-c con 0x7fec6013f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:15:16.147 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.144+0000 7fec64d58640 10 monclient(hunting): picked mon.noname-a con 0x7fec601430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:15:16.147 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.144+0000 7fec64d58640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec601db2f0 con 0x7fec601430c0 2024-03-25T18:15:16.147 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.144+0000 7fec64d58640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec60139180 con 0x7fec600a98f0 2024-03-25T18:15:16.147 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.144+0000 7fec64d58640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec60137670 con 0x7fec6013f710 2024-03-25T18:15:16.147 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.144+0000 7fec64d58640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:16.147 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.144+0000 7fec64d58640 10 monclient(hunting): authenticate will time out at 7397.455105s 2024-03-25T18:15:16.161 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.160+0000 7fec5e59c640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fec600a98f0 0x7fec601dabe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:35626/0 (socket says 172.21.15.156:35626) 2024-03-25T18:15:16.161 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.160+0000 7fec5e59c640 1 -- 172.21.15.156:0/2635507674 learned_addr learned my addr 172.21.15.156:0/2635507674 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:15:16.164 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.162+0000 7fec56ffd640 10 client.?.objecter ms_handle_connect 0x7fec600a98f0 2024-03-25T18:15:16.164 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.163+0000 7fec56ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:16.174 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.173+0000 7fec56ffd640 10 client.?.objecter ms_handle_connect 0x7fec6013f710 2024-03-25T18:15:16.174 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.173+0000 7fec56ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:16.180 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.179+0000 7fec56ffd640 1 -- 172.21.15.156:0/2635507674 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3710463783 0 0) 0x7fec60139180 con 0x7fec600a98f0 2024-03-25T18:15:16.180 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.179+0000 7fec56ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:16.180 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.179+0000 7fec56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:16.180 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.179+0000 7fec56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:16.180 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.179+0000 7fec56ffd640 10 monclient(hunting): my global_id is 4540 2024-03-25T18:15:16.180 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.179+0000 7fec56ffd640 1 -- 172.21.15.156:0/2635507674 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec40004eb0 con 0x7fec600a98f0 2024-03-25T18:15:16.182 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.181+0000 7fec56ffd640 10 client.?.objecter ms_handle_connect 0x7fec601430c0 2024-03-25T18:15:16.182 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.181+0000 7fec56ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:16.183 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.182+0000 7fec56ffd640 1 -- 172.21.15.156:0/2635507674 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154888164 0 0) 0x7fec60137670 con 0x7fec6013f710 2024-03-25T18:15:16.183 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.182+0000 7fec56ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:16.183 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.182+0000 7fec56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:16.183 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.182+0000 7fec56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:16.183 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.182+0000 7fec56ffd640 10 monclient(hunting): my global_id is 4544 2024-03-25T18:15:16.183 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.182+0000 7fec56ffd640 1 -- 172.21.15.156:0/2635507674 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec60139180 con 0x7fec6013f710 2024-03-25T18:15:16.197 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.196+0000 7fec56ffd640 1 -- 172.21.15.156:0/2635507674 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2236361760 0 0) 0x7fec601db2f0 con 0x7fec601430c0 2024-03-25T18:15:16.197 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.196+0000 7fec56ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:16.197 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.196+0000 7fec56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:16.198 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.196+0000 7fec56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:16.198 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.196+0000 7fec56ffd640 10 monclient(hunting): my global_id is 4533 2024-03-25T18:15:16.198 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.196+0000 7fec56ffd640 1 -- 172.21.15.156:0/2635507674 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec60137670 con 0x7fec601430c0 2024-03-25T18:15:16.198 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.196+0000 7fec56ffd640 1 -- 172.21.15.156:0/2635507674 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3419606568 0 0) 0x7fec60139180 con 0x7fec6013f710 2024-03-25T18:15:16.198 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.196+0000 7fec56ffd640 1 -- 172.21.15.156:0/2635507674 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec601db2f0 con 0x7fec6013f710 2024-03-25T18:15:16.198 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.196+0000 7fec56ffd640 1 -- 172.21.15.156:0/2635507674 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fec480018a0 con 0x7fec6013f710 2024-03-25T18:15:16.198 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.196+0000 7fec56ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:16.198 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.196+0000 7fec56ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:16.198 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.196+0000 7fec56ffd640 10 monclient(hunting): dump: 2024-03-25T18:15:16.198 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:16.198 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:16.198 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:16.198 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:16.199 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:16.199 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:16.199 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:16.199 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:16.199 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:16.199 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:16.199 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.196+0000 7fec56ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:16.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.198+0000 7fec56ffd640 1 -- 172.21.15.156:0/2635507674 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1038875779 0 0) 0x7fec40004eb0 con 0x7fec600a98f0 2024-03-25T18:15:16.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.199+0000 7fec56ffd640 1 -- 172.21.15.156:0/2635507674 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec60139180 con 0x7fec600a98f0 2024-03-25T18:15:16.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.199+0000 7fec56ffd640 1 -- 172.21.15.156:0/2635507674 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fec50001690 con 0x7fec600a98f0 2024-03-25T18:15:16.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.199+0000 7fec56ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:16.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.199+0000 7fec56ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:16.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.199+0000 7fec56ffd640 10 monclient(hunting): dump: 2024-03-25T18:15:16.200 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:16.200 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:16.200 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:16.200 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:16.200 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:16.201 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:16.201 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:16.201 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:16.201 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:16.201 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:16.229 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.227+0000 7fec56ffd640 1 -- 172.21.15.156:0/2635507674 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4249506566 0 0) 0x7fec60139180 con 0x7fec600a98f0 2024-03-25T18:15:16.229 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.227+0000 7fec56ffd640 1 -- 172.21.15.156:0/2635507674 >> v1:172.21.15.156:6790/0 conn(0x7fec6013f710 legacy=0x7fec60132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:16.229 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.227+0000 7fec56ffd640 1 -- 172.21.15.156:0/2635507674 >> v1:172.21.15.156:6789/0 conn(0x7fec601430c0 legacy=0x7fec601e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:16.229 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.227+0000 7fec56ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:16.229 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.227+0000 7fec56ffd640 1 monclient: found mon.b 2024-03-25T18:15:16.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.227+0000 7fec56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:16.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.227+0000 7fec56ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:15:16.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.227+0000 7fec56ffd640 1 -- 172.21.15.156:0/2635507674 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec60137200 con 0x7fec600a98f0 2024-03-25T18:15:16.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.227+0000 7fec56ffd640 10 monclient: _finish_auth 0 2024-03-25T18:15:16.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.227+0000 7fec56ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:16.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.227+0000 7fec56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:16.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.227+0000 7fec64d58640 5 monclient: authenticate success, global_id 4540 2024-03-25T18:15:16.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.227+0000 7fec64d58640 10 monclient: _renew_subs 2024-03-25T18:15:16.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.227+0000 7fec64d58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:15:16.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.227+0000 7fec64d58640 1 -- 172.21.15.156:0/2635507674 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec600b7290 con 0x7fec600a98f0 2024-03-25T18:15:16.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.227+0000 7fec64d58640 10 client.4540.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:16.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.227+0000 7fec64d58640 10 monclient: _renew_subs 2024-03-25T18:15:16.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.227+0000 7fec64d58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:15:16.230 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.227+0000 7fec64d58640 1 -- 172.21.15.156:0/2635507674 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec60133680 con 0x7fec600a98f0 2024-03-25T18:15:16.231 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.227+0000 7fec64d58640 1 librados: init done 2024-03-25T18:15:16.231 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.228+0000 7fec64d58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:16.231 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.228+0000 7fec64d58640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:16.231 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.228+0000 7fec64d58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:15:16.231 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.228+0000 7fec64d58640 1 -- 172.21.15.156:0/2635507674 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec60137800 con 0x7fec600a98f0 2024-03-25T18:15:16.510 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.509+0000 7fec56ffd640 1 -- 172.21.15.156:0/2635507674 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec500038a0 con 0x7fec600a98f0 2024-03-25T18:15:16.510 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.509+0000 7fec56ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:16.510 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.509+0000 7fec56ffd640 1 -- 172.21.15.156:0/2635507674 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fec50003c70 con 0x7fec600a98f0 2024-03-25T18:15:16.510 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.509+0000 7fec56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:16.510 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.509+0000 7fec56ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:16.510 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.509+0000 7fec56ffd640 10 monclient: dump: 2024-03-25T18:15:16.510 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:16.510 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:16.510 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:16.510 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:16.511 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:16.511 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:16.511 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:16.511 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:16.511 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:16.511 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:16.511 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.509+0000 7fec5d59a640 4 set_mon_vals no callback set 2024-03-25T18:15:16.511 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.510+0000 7fec56ffd640 1 -- 172.21.15.156:0/2635507674 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7fec50003e50 con 0x7fec600a98f0 2024-03-25T18:15:16.512 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.510+0000 7fec57fff640 10 monclient: get_auth_request con 0x7fec4006dfb0 auth_method 0 2024-03-25T18:15:16.512 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.510+0000 7fec56ffd640 10 client.4540.objecter ms_handle_connect 0x7fec4006dfb0 2024-03-25T18:15:16.516 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.515+0000 7fec56ffd640 1 -- 172.21.15.156:0/2635507674 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7fec500549b0 con 0x7fec600a98f0 2024-03-25T18:15:16.517 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.515+0000 7fec56ffd640 10 client.4540.objecter ms_dispatch 0x7fec60000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:16.517 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.515+0000 7fec56ffd640 3 client.4540.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:16.517 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.515+0000 7fec56ffd640 3 client.4540.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:16.517 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.515+0000 7fec56ffd640 20 client.4540.objecter dump_active .. 0 homeless 2024-03-25T18:15:16.517 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.515+0000 7fec56ffd640 1 -- 172.21.15.156:0/2635507674 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fec500504c0 con 0x7fec600a98f0 2024-03-25T18:15:16.517 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.515+0000 7fec56ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:16.517 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.515+0000 7fec56ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:16.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.738+0000 7fec64d58640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:16.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.738+0000 7fec64d58640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:16.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.738+0000 7fec64d58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:15:16.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.738+0000 7fec64d58640 1 -- 172.21.15.156:0/2635507674 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec600627b0 con 0x7fec600a98f0 2024-03-25T18:15:16.773 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.772+0000 7fec56ffd640 1 -- 172.21.15.156:0/2635507674 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7fec500523f0 con 0x7fec600a98f0 2024-03-25T18:15:16.774 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.772+0000 7fec56ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:16.774 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.772+0000 7fec56ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:16.774 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:15:16.774 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:16.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.775+0000 7fec64d58640 10 librados: watch_flush enter 2024-03-25T18:15:16.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.775+0000 7fec64d58640 10 librados: watch_flush exit 2024-03-25T18:15:16.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.775+0000 7fec64d58640 20 client.4540.objecter shutdown clearing up homeless session... 2024-03-25T18:15:16.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.775+0000 7fec64d58640 10 client.4540.objecter successfully canceled tick 2024-03-25T18:15:16.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.775+0000 7fec64d58640 1 -- 172.21.15.156:0/2635507674 >> v1:172.21.15.194:6800/2946427713 conn(0x7fec4006dfb0 legacy=0x7fec40070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:16.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.775+0000 7fec64d58640 10 monclient: shutdown 2024-03-25T18:15:16.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.775+0000 7fec64d58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:16.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.775+0000 7fec64d58640 1 -- 172.21.15.156:0/2635507674 >> v1:172.21.15.194:6789/0 conn(0x7fec600a98f0 legacy=0x7fec601dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:16.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.775+0000 7fec64d58640 1 -- 172.21.15.156:0/2635507674 shutdown_connections 2024-03-25T18:15:16.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.775+0000 7fec64d58640 1 -- 172.21.15.156:0/2635507674 >> 172.21.15.156:0/2635507674 conn(0x7fec600b7ea0 msgr2=0x7fec60074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:16.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.775+0000 7fec64d58640 1 -- 172.21.15.156:0/2635507674 shutdown_connections 2024-03-25T18:15:16.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.775+0000 7fec64d58640 1 -- 172.21.15.156:0/2635507674 wait complete. 2024-03-25T18:15:16.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:16.775+0000 7fec64d58640 1 librados: shutdown 2024-03-25T18:15:16.789 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:19.420 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:07:25.410 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:19.476 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:07:25.466 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:19.511 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:07:25.501 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:19.519 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:07:25.486 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:19.532 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:07:25.499 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:19.557 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:07:25.547 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:19.568 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:07:25.558 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:19.570 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:07:25.537 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:19.591 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:07:25.582 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:23.596 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:23.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.709+0000 7f9716f64640 10 monclient: get_monmap_and_config 2024-03-25T18:15:23.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.709+0000 7f9716f64640 10 monclient: build_initial_monmap 2024-03-25T18:15:23.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.709+0000 7f9716f64640 10 monclient: monmap: 2024-03-25T18:15:23.710 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:15:23.710 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:23.710 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:15:23.710405+0000 2024-03-25T18:15:23.710 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:15:23.710405+0000 2024-03-25T18:15:23.711 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:23.711 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:23.711 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:15:23.711 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:15:23.711 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:15:23.711 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:23.712 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.710+0000 7f9716f64640 1 Processor -- start 2024-03-25T18:15:23.712 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.710+0000 7f9716f64640 1 -- start start 2024-03-25T18:15:23.712 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.710+0000 7f9716f64640 10 monclient: init 2024-03-25T18:15:23.712 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.711+0000 7f9716f64640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:23.712 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.711+0000 7f9716f64640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:23.712 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.711+0000 7f9716f64640 10 monclient(hunting): picked mon.noname-a con 0x7f97100b4d60 addr v1:172.21.15.156:6789/0 2024-03-25T18:15:23.713 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.711+0000 7f9716f64640 10 monclient(hunting): picked mon.noname-c con 0x7f97100b1730 addr v1:172.21.15.194:6789/0 2024-03-25T18:15:23.713 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.711+0000 7f9716f64640 10 monclient(hunting): picked mon.noname-b con 0x7f971014be90 addr v1:172.21.15.156:6790/0 2024-03-25T18:15:23.713 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.711+0000 7f9716f64640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97100af940 con 0x7f97100b4d60 2024-03-25T18:15:23.713 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.711+0000 7f9716f64640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97100b0a70 con 0x7f97100b1730 2024-03-25T18:15:23.713 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.711+0000 7f9716f64640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97100b0d50 con 0x7f971014be90 2024-03-25T18:15:23.713 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.711+0000 7f9716f64640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:23.713 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.711+0000 7f9716f64640 10 monclient(hunting): authenticate will time out at 7405.022005s 2024-03-25T18:15:23.720 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.719+0000 7f970ffff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f97100b1730 0x7f97100b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:56612/0 (socket says 172.21.15.156:56612) 2024-03-25T18:15:23.720 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.719+0000 7f970ffff640 1 -- 172.21.15.156:0/3523020450 learned_addr learned my addr 172.21.15.156:0/3523020450 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:15:23.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.730+0000 7f970f7fe640 1 -- 172.21.15.156:0/3523020450 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1068026844 0 0) 0x7f97100b0a70 con 0x7f97100b1730 2024-03-25T18:15:23.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.730+0000 7f970f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:23.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.730+0000 7f970f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:23.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.730+0000 7f970f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:23.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.730+0000 7f970f7fe640 10 monclient(hunting): my global_id is 4543 2024-03-25T18:15:23.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.730+0000 7f970f7fe640 1 -- 172.21.15.156:0/3523020450 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9700003520 con 0x7f97100b1730 2024-03-25T18:15:23.739 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.738+0000 7f970f7fe640 1 -- 172.21.15.156:0/3523020450 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3138859839 0 0) 0x7f9700003520 con 0x7f97100b1730 2024-03-25T18:15:23.739 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.738+0000 7f970f7fe640 1 -- 172.21.15.156:0/3523020450 >> v1:172.21.15.156:6790/0 conn(0x7f971014be90 legacy=0x7f971014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:23.739 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.738+0000 7f970f7fe640 1 -- 172.21.15.156:0/3523020450 >> v1:172.21.15.156:6789/0 conn(0x7f97100b4d60 legacy=0x7f971013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:23.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.738+0000 7f970f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:15:23.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.738+0000 7f970f7fe640 1 monclient: found mon.noname-c 2024-03-25T18:15:23.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.738+0000 7f970f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:23.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.738+0000 7f970f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:15:23.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.738+0000 7f970f7fe640 1 -- 172.21.15.156:0/3523020450 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97100b1030 con 0x7f97100b1730 2024-03-25T18:15:23.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.738+0000 7f970f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:15:23.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.738+0000 7f970f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:15:23.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.738+0000 7f970f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:23.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.738+0000 7f970f7fe640 1 -- 172.21.15.156:0/3523020450 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f96fc001690 con 0x7f97100b1730 2024-03-25T18:15:23.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.738+0000 7f9716f64640 5 monclient: authenticate success, global_id 4543 2024-03-25T18:15:23.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.738+0000 7f9716f64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:23.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.738+0000 7f970f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:23.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.738+0000 7f970f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:23.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.738+0000 7f970f7fe640 10 monclient: dump: 2024-03-25T18:15:23.740 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:23.741 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:23.741 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:23.741 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:23.741 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:23.741 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:23.741 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:23.741 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:23.741 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:23.741 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:23.741 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.738+0000 7f9716f64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:23.748 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.747+0000 7f970f7fe640 1 -- 172.21.15.156:0/3523020450 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96fc0017f0 con 0x7f97100b1730 2024-03-25T18:15:23.748 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.747+0000 7f970f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:23.748 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.747+0000 7f970f7fe640 1 -- 172.21.15.156:0/3523020450 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f96fc001c00 con 0x7f97100b1730 2024-03-25T18:15:23.748 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.747+0000 7f9716f64640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:23.748 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.747+0000 7f9716f64640 4 set_mon_vals no callback set 2024-03-25T18:15:23.748 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.747+0000 7f970f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:23.748 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.747+0000 7f970f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:23.748 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.747+0000 7f970f7fe640 10 monclient: dump: 2024-03-25T18:15:23.748 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:23.748 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:23.749 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:23.749 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:23.749 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:23.749 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:23.749 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:23.749 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:23.749 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:23.749 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:23.750 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.748+0000 7f9716f64640 10 monclient: shutdown 2024-03-25T18:15:23.750 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.748+0000 7f9716f64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:23.751 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.748+0000 7f9716f64640 1 -- 172.21.15.156:0/3523020450 >> v1:172.21.15.194:6789/0 conn(0x7f97100b1730 legacy=0x7f97100b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:23.751 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.748+0000 7f9716f64640 1 -- 172.21.15.156:0/3523020450 shutdown_connections 2024-03-25T18:15:23.751 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.748+0000 7f9716f64640 1 -- 172.21.15.156:0/3523020450 >> 172.21.15.156:0/3523020450 conn(0x7f97100617a0 msgr2=0x7f9710061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:23.751 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.748+0000 7f9716f64640 1 -- 172.21.15.156:0/3523020450 shutdown_connections 2024-03-25T18:15:23.751 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.748+0000 7f9716f64640 1 -- 172.21.15.156:0/3523020450 wait complete. 2024-03-25T18:15:23.751 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.748+0000 7f9716f64640 10 monclient: build_initial_monmap 2024-03-25T18:15:23.751 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.748+0000 7f9716f64640 10 monclient: monmap: 2024-03-25T18:15:23.751 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:15:23.751 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:23.751 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:15:23.751 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:15:23.751 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:23.751 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:15:23.751 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:15:23.752 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:15:23.752 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:15:23.752 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:23.752 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.749+0000 7f9716f64640 1 librados: starting msgr at 2024-03-25T18:15:23.752 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.749+0000 7f9716f64640 1 librados: starting objecter 2024-03-25T18:15:23.752 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.749+0000 7f9716f64640 1 Processor -- start 2024-03-25T18:15:23.752 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.749+0000 7f9716f64640 1 -- start start 2024-03-25T18:15:23.752 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.749+0000 7f9716f64640 1 librados: setting wanted keys 2024-03-25T18:15:23.752 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.749+0000 7f9716f64640 1 librados: calling monclient init 2024-03-25T18:15:23.752 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.749+0000 7f9716f64640 10 monclient: init 2024-03-25T18:15:23.753 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.750+0000 7f9716f64640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:23.753 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.750+0000 7f9716f64640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:23.753 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.750+0000 7f9716f64640 10 monclient(hunting): picked mon.noname-a con 0x7f97100b1730 addr v1:172.21.15.156:6789/0 2024-03-25T18:15:23.754 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.750+0000 7f9716f64640 10 monclient(hunting): picked mon.noname-b con 0x7f97100b4d60 addr v1:172.21.15.194:6789/0 2024-03-25T18:15:23.754 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.750+0000 7f9716f64640 10 monclient(hunting): picked mon.noname-c con 0x7f971014be90 addr v1:172.21.15.156:6790/0 2024-03-25T18:15:23.754 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.750+0000 7f9716f64640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f971013bd10 con 0x7f97100b1730 2024-03-25T18:15:23.754 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.750+0000 7f9716f64640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9710141c30 con 0x7f97100b4d60 2024-03-25T18:15:23.754 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.750+0000 7f9716f64640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9710141f10 con 0x7f971014be90 2024-03-25T18:15:23.754 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.750+0000 7f9716f64640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:23.754 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.750+0000 7f9716f64640 10 monclient(hunting): authenticate will time out at 7405.061384s 2024-03-25T18:15:23.771 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.770+0000 7f9714d00640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f97100b1730 0x7f97101341f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:59834/0 (socket says 172.21.15.156:59834) 2024-03-25T18:15:23.771 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.770+0000 7f9714d00640 1 -- 172.21.15.156:0/3625616007 learned_addr learned my addr 172.21.15.156:0/3625616007 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:15:23.773 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.772+0000 7f970d7fa640 10 client.?.objecter ms_handle_connect 0x7f971014be90 2024-03-25T18:15:23.773 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.772+0000 7f970d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:23.774 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.773+0000 7f970d7fa640 10 client.?.objecter ms_handle_connect 0x7f97100b1730 2024-03-25T18:15:23.774 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.773+0000 7f970d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:23.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.776+0000 7f970d7fa640 10 client.?.objecter ms_handle_connect 0x7f97100b4d60 2024-03-25T18:15:23.777 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.776+0000 7f970d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:23.780 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.779+0000 7f970d7fa640 1 -- 172.21.15.156:0/3625616007 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698772478 0 0) 0x7f971013bd10 con 0x7f97100b1730 2024-03-25T18:15:23.780 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.779+0000 7f970d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:23.780 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.779+0000 7f970d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:23.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.779+0000 7f970d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:23.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.779+0000 7f970d7fa640 10 monclient(hunting): my global_id is 4539 2024-03-25T18:15:23.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.779+0000 7f970d7fa640 1 -- 172.21.15.156:0/3625616007 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96f4004eb0 con 0x7f97100b1730 2024-03-25T18:15:23.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.780+0000 7f970d7fa640 1 -- 172.21.15.156:0/3625616007 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2758039039 0 0) 0x7f9710141c30 con 0x7f97100b4d60 2024-03-25T18:15:23.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.780+0000 7f970d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:23.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.780+0000 7f970d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:23.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.780+0000 7f970d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:23.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.780+0000 7f970d7fa640 10 monclient(hunting): my global_id is 4546 2024-03-25T18:15:23.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.780+0000 7f970d7fa640 1 -- 172.21.15.156:0/3625616007 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f971013bd10 con 0x7f97100b4d60 2024-03-25T18:15:23.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.780+0000 7f970d7fa640 1 -- 172.21.15.156:0/3625616007 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812846237 0 0) 0x7f9710141f10 con 0x7f971014be90 2024-03-25T18:15:23.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.780+0000 7f970d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:23.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.780+0000 7f970d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:23.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.780+0000 7f970d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:23.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.780+0000 7f970d7fa640 10 monclient(hunting): my global_id is 4550 2024-03-25T18:15:23.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.780+0000 7f970d7fa640 1 -- 172.21.15.156:0/3625616007 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9710141c30 con 0x7f971014be90 2024-03-25T18:15:23.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.794+0000 7f970d7fa640 1 -- 172.21.15.156:0/3625616007 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3959062469 0 0) 0x7f971013bd10 con 0x7f97100b4d60 2024-03-25T18:15:23.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.794+0000 7f970d7fa640 1 -- 172.21.15.156:0/3625616007 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9710141f10 con 0x7f97100b4d60 2024-03-25T18:15:23.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.794+0000 7f970d7fa640 1 -- 172.21.15.156:0/3625616007 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f96fc001b00 con 0x7f97100b4d60 2024-03-25T18:15:23.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.794+0000 7f970d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:23.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.794+0000 7f970d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:23.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.794+0000 7f970d7fa640 10 monclient(hunting): dump: 2024-03-25T18:15:23.795 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:23.796 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:23.796 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:23.796 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:23.796 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:23.796 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:23.796 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:23.796 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:23.796 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:23.796 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:23.796 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.794+0000 7f970d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:23.799 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.798+0000 7f970d7fa640 1 -- 172.21.15.156:0/3625616007 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2767026854 0 0) 0x7f96f4004eb0 con 0x7f97100b1730 2024-03-25T18:15:23.799 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.798+0000 7f970d7fa640 1 -- 172.21.15.156:0/3625616007 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f971013bd10 con 0x7f97100b1730 2024-03-25T18:15:23.799 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.798+0000 7f970d7fa640 1 -- 172.21.15.156:0/3625616007 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9704001a30 con 0x7f97100b1730 2024-03-25T18:15:23.799 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.798+0000 7f970d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:23.799 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.798+0000 7f970d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:23.799 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.798+0000 7f970d7fa640 10 monclient(hunting): dump: 2024-03-25T18:15:23.799 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:23.799 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:23.799 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:23.799 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:23.799 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:23.799 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:23.800 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:23.800 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:23.800 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:23.800 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:23.802 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.801+0000 7f970d7fa640 1 -- 172.21.15.156:0/3625616007 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 339881050 0 0) 0x7f9710141c30 con 0x7f971014be90 2024-03-25T18:15:23.802 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.801+0000 7f970d7fa640 1 -- 172.21.15.156:0/3625616007 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96f4004eb0 con 0x7f971014be90 2024-03-25T18:15:23.802 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.801+0000 7f970d7fa640 1 -- 172.21.15.156:0/3625616007 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9708001a90 con 0x7f971014be90 2024-03-25T18:15:23.802 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.801+0000 7f970d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:23.802 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.801+0000 7f970d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:23.802 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.801+0000 7f970d7fa640 10 monclient(hunting): dump: 2024-03-25T18:15:23.802 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:23.802 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:23.803 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:23.803 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:23.803 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:23.803 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:23.803 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:23.803 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:23.803 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:23.803 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:23.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.807+0000 7f970d7fa640 1 -- 172.21.15.156:0/3625616007 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1162327313 0 0) 0x7f9710141f10 con 0x7f97100b4d60 2024-03-25T18:15:23.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.807+0000 7f970d7fa640 1 -- 172.21.15.156:0/3625616007 >> v1:172.21.15.156:6790/0 conn(0x7f971014be90 legacy=0x7f97101e35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:23.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.807+0000 7f970d7fa640 1 -- 172.21.15.156:0/3625616007 >> v1:172.21.15.156:6789/0 conn(0x7f97100b1730 legacy=0x7f97101341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:23.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.807+0000 7f970d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:15:23.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.807+0000 7f970d7fa640 1 monclient: found mon.b 2024-03-25T18:15:23.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.807+0000 7f970d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:23.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.807+0000 7f970d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:15:23.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.807+0000 7f970d7fa640 1 -- 172.21.15.156:0/3625616007 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9710140810 con 0x7f97100b4d60 2024-03-25T18:15:23.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.807+0000 7f970d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:15:23.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.807+0000 7f970d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:15:23.810 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.807+0000 7f970d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:23.810 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.807+0000 7f9716f64640 5 monclient: authenticate success, global_id 4546 2024-03-25T18:15:23.810 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.807+0000 7f9716f64640 10 monclient: _renew_subs 2024-03-25T18:15:23.810 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.807+0000 7f9716f64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:15:23.810 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.807+0000 7f9716f64640 1 -- 172.21.15.156:0/3625616007 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9710140d30 con 0x7f97100b4d60 2024-03-25T18:15:23.810 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.807+0000 7f9716f64640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:23.810 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.807+0000 7f9716f64640 10 monclient: _renew_subs 2024-03-25T18:15:23.810 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.807+0000 7f9716f64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:15:23.810 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.807+0000 7f9716f64640 1 -- 172.21.15.156:0/3625616007 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9710140560 con 0x7f97100b4d60 2024-03-25T18:15:23.810 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.807+0000 7f9716f64640 1 librados: init done 2024-03-25T18:15:23.811 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.808+0000 7f9716f64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:23.811 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.808+0000 7f9716f64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:23.811 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.808+0000 7f9716f64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:15:23.811 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:23.808+0000 7f9716f64640 1 -- 172.21.15.156:0/3625616007 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97100627b0 con 0x7f97100b4d60 2024-03-25T18:15:24.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.025+0000 7f970d7fa640 1 -- 172.21.15.156:0/3625616007 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96fc001ca0 con 0x7f97100b4d60 2024-03-25T18:15:24.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.025+0000 7f970d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:24.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.025+0000 7f970effd640 4 set_mon_vals no callback set 2024-03-25T18:15:24.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.093+0000 7f970d7fa640 1 -- 172.21.15.156:0/3625616007 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f96fc004c30 con 0x7f97100b4d60 2024-03-25T18:15:24.094 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.093+0000 7f970d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:24.094 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.093+0000 7f970d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:24.094 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.093+0000 7f970d7fa640 10 monclient: dump: 2024-03-25T18:15:24.094 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:24.094 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:24.094 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:24.094 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:24.094 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:24.094 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:24.094 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:24.094 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:24.094 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:24.094 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:24.095 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.093+0000 7f970d7fa640 1 -- 172.21.15.156:0/3625616007 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f96fc01ac00 con 0x7f97100b4d60 2024-03-25T18:15:24.095 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.094+0000 7f9714d00640 10 monclient: get_auth_request con 0x7f96f406e0f0 auth_method 0 2024-03-25T18:15:24.095 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.094+0000 7f970d7fa640 10 client.4546.objecter ms_handle_connect 0x7f96f406e0f0 2024-03-25T18:15:24.099 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.097+0000 7f970d7fa640 1 -- 172.21.15.156:0/3625616007 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f96fc054840 con 0x7f97100b4d60 2024-03-25T18:15:24.099 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.097+0000 7f970d7fa640 10 client.4546.objecter ms_dispatch 0x7f9710000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:24.099 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.097+0000 7f970d7fa640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:24.099 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.097+0000 7f970d7fa640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:24.099 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.097+0000 7f970d7fa640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-25T18:15:24.103 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.102+0000 7f970d7fa640 1 -- 172.21.15.156:0/3625616007 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f96fc050350 con 0x7f97100b4d60 2024-03-25T18:15:24.103 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.102+0000 7f970d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:24.103 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.102+0000 7f970d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:24.356 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.355+0000 7f9716f64640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:24.356 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.355+0000 7f9716f64640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:24.357 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.355+0000 7f9716f64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:15:24.357 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.355+0000 7f9716f64640 1 -- 172.21.15.156:0/3625616007 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97101409a0 con 0x7f97100b4d60 2024-03-25T18:15:24.490 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.489+0000 7f970d7fa640 1 -- 172.21.15.156:0/3625616007 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f96fc052280 con 0x7f97100b4d60 2024-03-25T18:15:24.491 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.489+0000 7f970d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:24.491 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.489+0000 7f970d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:24.491 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:15:24.491 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:24.493 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.491+0000 7f9716f64640 10 librados: watch_flush enter 2024-03-25T18:15:24.493 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.491+0000 7f9716f64640 10 librados: watch_flush exit 2024-03-25T18:15:24.493 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.491+0000 7f9716f64640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-25T18:15:24.493 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.491+0000 7f9716f64640 10 client.4546.objecter successfully canceled tick 2024-03-25T18:15:24.493 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.492+0000 7f9716f64640 1 -- 172.21.15.156:0/3625616007 >> v1:172.21.15.194:6800/2946427713 conn(0x7f96f406e0f0 legacy=0x7f96f40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:24.494 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.492+0000 7f9716f64640 10 monclient: shutdown 2024-03-25T18:15:24.494 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.492+0000 7f9716f64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:24.494 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.492+0000 7f9716f64640 1 -- 172.21.15.156:0/3625616007 >> v1:172.21.15.194:6789/0 conn(0x7f97100b4d60 legacy=0x7f9710142330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:24.494 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.492+0000 7f9716f64640 1 -- 172.21.15.156:0/3625616007 shutdown_connections 2024-03-25T18:15:24.494 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.492+0000 7f9716f64640 1 -- 172.21.15.156:0/3625616007 >> 172.21.15.156:0/3625616007 conn(0x7f97100617a0 msgr2=0x7f97100b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:24.494 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.492+0000 7f9716f64640 1 -- 172.21.15.156:0/3625616007 shutdown_connections 2024-03-25T18:15:24.494 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.492+0000 7f9716f64640 1 -- 172.21.15.156:0/3625616007 wait complete. 2024-03-25T18:15:24.494 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:24.492+0000 7f9716f64640 1 librados: shutdown 2024-03-25T18:15:24.505 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:31.311 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:31.420 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.419+0000 7f3ba9546640 10 monclient: get_monmap_and_config 2024-03-25T18:15:31.420 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.419+0000 7f3ba9546640 10 monclient: build_initial_monmap 2024-03-25T18:15:31.421 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.419+0000 7f3ba9546640 10 monclient: monmap: 2024-03-25T18:15:31.421 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:15:31.421 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:31.421 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:15:31.420638+0000 2024-03-25T18:15:31.421 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:15:31.420638+0000 2024-03-25T18:15:31.421 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:31.421 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:31.421 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:15:31.421 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:15:31.421 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:15:31.421 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:31.422 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.420+0000 7f3ba9546640 1 Processor -- start 2024-03-25T18:15:31.422 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.420+0000 7f3ba9546640 1 -- start start 2024-03-25T18:15:31.422 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.420+0000 7f3ba9546640 10 monclient: init 2024-03-25T18:15:31.422 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.420+0000 7f3ba9546640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:31.423 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.420+0000 7f3ba9546640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:15:31.423 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.420+0000 7f3ba9546640 10 monclient(hunting): picked mon.noname-c con 0x7f3ba40a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:15:31.423 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.420+0000 7f3ba9546640 10 monclient(hunting): picked mon.noname-b con 0x7f3ba413f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:15:31.423 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.420+0000 7f3ba9546640 10 monclient(hunting): picked mon.noname-a con 0x7f3ba41430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:15:31.423 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.420+0000 7f3ba9546640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ba4142520 con 0x7f3ba41430c0 2024-03-25T18:15:31.423 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.421+0000 7f3ba9546640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ba40b28a0 con 0x7f3ba40a98f0 2024-03-25T18:15:31.423 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.421+0000 7f3ba9546640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ba40b2a30 con 0x7f3ba413f710 2024-03-25T18:15:31.423 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.421+0000 7f3ba9546640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:31.423 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.421+0000 7f3ba9546640 10 monclient(hunting): authenticate will time out at 7412.731818s 2024-03-25T18:15:31.428 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.427+0000 7f3ba37fe640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f3ba41430c0 0x7f3ba414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:36294/0 (socket says 172.21.15.156:36294) 2024-03-25T18:15:31.428 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.427+0000 7f3ba37fe640 1 -- 172.21.15.156:0/980265762 learned_addr learned my addr 172.21.15.156:0/980265762 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:15:31.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.434+0000 7f3ba1ffb640 1 -- 172.21.15.156:0/980265762 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076574972 0 0) 0x7f3ba4142520 con 0x7f3ba41430c0 2024-03-25T18:15:31.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.434+0000 7f3ba1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:31.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.434+0000 7f3ba1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:31.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.434+0000 7f3ba1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:31.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.434+0000 7f3ba1ffb640 10 monclient(hunting): my global_id is 4542 2024-03-25T18:15:31.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.434+0000 7f3ba1ffb640 1 -- 172.21.15.156:0/980265762 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b8c003520 con 0x7f3ba41430c0 2024-03-25T18:15:31.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.442+0000 7f3ba1ffb640 1 -- 172.21.15.156:0/980265762 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1973488531 0 0) 0x7f3b8c003520 con 0x7f3ba41430c0 2024-03-25T18:15:31.444 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.442+0000 7f3ba1ffb640 1 -- 172.21.15.156:0/980265762 >> v1:172.21.15.156:6790/0 conn(0x7f3ba413f710 legacy=0x7f3ba41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:31.444 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.442+0000 7f3ba1ffb640 1 -- 172.21.15.156:0/980265762 >> v1:172.21.15.194:6789/0 conn(0x7f3ba40a98f0 legacy=0x7f3ba40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:31.444 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.442+0000 7f3ba1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:15:31.444 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.442+0000 7f3ba1ffb640 1 monclient: found mon.noname-a 2024-03-25T18:15:31.444 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.442+0000 7f3ba1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:31.444 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.442+0000 7f3ba1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:15:31.444 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.442+0000 7f3ba1ffb640 1 -- 172.21.15.156:0/980265762 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba40b44c0 con 0x7f3ba41430c0 2024-03-25T18:15:31.444 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.442+0000 7f3ba1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:15:31.444 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.442+0000 7f3ba1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:15:31.444 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.442+0000 7f3ba1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:31.444 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.442+0000 7f3ba1ffb640 1 -- 172.21.15.156:0/980265762 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f3b9c001a10 con 0x7f3ba41430c0 2024-03-25T18:15:31.444 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.442+0000 7f3ba1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:31.444 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.442+0000 7f3ba1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:31.444 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.442+0000 7f3ba1ffb640 10 monclient: dump: 2024-03-25T18:15:31.445 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:31.445 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:31.445 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:31.445 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:31.445 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:31.445 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:31.445 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:31.445 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:31.445 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:31.445 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:31.445 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.442+0000 7f3ba9546640 5 monclient: authenticate success, global_id 4542 2024-03-25T18:15:31.445 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.442+0000 7f3ba9546640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:31.455 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.453+0000 7f3ba1ffb640 1 -- 172.21.15.156:0/980265762 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b9c001bf0 con 0x7f3ba41430c0 2024-03-25T18:15:31.455 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.454+0000 7f3ba1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:31.455 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.454+0000 7f3ba1ffb640 1 -- 172.21.15.156:0/980265762 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f3b9c002000 con 0x7f3ba41430c0 2024-03-25T18:15:31.455 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.454+0000 7f3ba9546640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:31.455 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.454+0000 7f3ba9546640 4 set_mon_vals no callback set 2024-03-25T18:15:31.455 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.454+0000 7f3ba1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:31.455 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.454+0000 7f3ba1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:31.455 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.454+0000 7f3ba1ffb640 10 monclient: dump: 2024-03-25T18:15:31.455 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:31.455 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:31.455 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:31.455 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:31.455 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:31.455 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:31.455 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:31.455 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:31.456 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:31.456 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:31.457 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.454+0000 7f3ba9546640 10 monclient: shutdown 2024-03-25T18:15:31.457 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.454+0000 7f3ba9546640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:31.457 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.454+0000 7f3ba9546640 1 -- 172.21.15.156:0/980265762 >> v1:172.21.15.156:6789/0 conn(0x7f3ba41430c0 legacy=0x7f3ba414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:31.457 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.454+0000 7f3ba9546640 1 -- 172.21.15.156:0/980265762 shutdown_connections 2024-03-25T18:15:31.457 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.454+0000 7f3ba9546640 1 -- 172.21.15.156:0/980265762 >> 172.21.15.156:0/980265762 conn(0x7f3ba40b7ea0 msgr2=0x7f3ba40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:31.457 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.454+0000 7f3ba9546640 1 -- 172.21.15.156:0/980265762 shutdown_connections 2024-03-25T18:15:31.457 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.454+0000 7f3ba9546640 1 -- 172.21.15.156:0/980265762 wait complete. 2024-03-25T18:15:31.457 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.455+0000 7f3ba9546640 10 monclient: build_initial_monmap 2024-03-25T18:15:31.457 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.455+0000 7f3ba9546640 10 monclient: monmap: 2024-03-25T18:15:31.457 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:15:31.457 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:31.458 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:15:31.458 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:15:31.458 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:31.458 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:15:31.458 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:15:31.458 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:15:31.458 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:15:31.458 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:31.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.455+0000 7f3ba9546640 1 librados: starting msgr at 2024-03-25T18:15:31.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.455+0000 7f3ba9546640 1 librados: starting objecter 2024-03-25T18:15:31.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.456+0000 7f3ba9546640 1 Processor -- start 2024-03-25T18:15:31.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.456+0000 7f3ba9546640 1 -- start start 2024-03-25T18:15:31.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.456+0000 7f3ba9546640 1 librados: setting wanted keys 2024-03-25T18:15:31.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.456+0000 7f3ba9546640 1 librados: calling monclient init 2024-03-25T18:15:31.459 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.456+0000 7f3ba9546640 10 monclient: init 2024-03-25T18:15:31.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.456+0000 7f3ba9546640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:31.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.456+0000 7f3ba9546640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:15:31.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.456+0000 7f3ba9546640 10 monclient(hunting): picked mon.noname-c con 0x7f3ba40a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:15:31.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.456+0000 7f3ba9546640 10 monclient(hunting): picked mon.noname-b con 0x7f3ba413f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:15:31.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.456+0000 7f3ba9546640 10 monclient(hunting): picked mon.noname-a con 0x7f3ba41430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:15:31.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.457+0000 7f3ba9546640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ba4138b60 con 0x7f3ba41430c0 2024-03-25T18:15:31.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.457+0000 7f3ba9546640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ba4137e40 con 0x7f3ba413f710 2024-03-25T18:15:31.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.457+0000 7f3ba9546640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ba41d82f0 con 0x7f3ba40a98f0 2024-03-25T18:15:31.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.457+0000 7f3ba9546640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:31.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.457+0000 7f3ba9546640 10 monclient(hunting): authenticate will time out at 7412.767878s 2024-03-25T18:15:31.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.459+0000 7f3ba37fe640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f3ba41430c0 0x7f3ba4133f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:36320/0 (socket says 172.21.15.156:36320) 2024-03-25T18:15:31.460 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.459+0000 7f3ba37fe640 1 -- 172.21.15.156:0/905291311 learned_addr learned my addr 172.21.15.156:0/905291311 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:15:31.461 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.460+0000 7f3b8b7fe640 10 client.?.objecter ms_handle_connect 0x7f3ba41430c0 2024-03-25T18:15:31.461 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.460+0000 7f3b8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:31.471 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.470+0000 7f3b8b7fe640 10 client.?.objecter ms_handle_connect 0x7f3ba40a98f0 2024-03-25T18:15:31.471 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.470+0000 7f3b8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:31.473 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.472+0000 7f3b8b7fe640 1 -- 172.21.15.156:0/905291311 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2160449668 0 0) 0x7f3ba4138b60 con 0x7f3ba41430c0 2024-03-25T18:15:31.473 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.472+0000 7f3b8b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:31.473 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.472+0000 7f3b8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:31.473 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.472+0000 7f3b8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:31.473 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.472+0000 7f3b8b7fe640 10 monclient(hunting): my global_id is 4545 2024-03-25T18:15:31.473 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.472+0000 7f3b8b7fe640 1 -- 172.21.15.156:0/905291311 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b90004eb0 con 0x7f3ba41430c0 2024-03-25T18:15:31.473 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.473+0000 7f3b8b7fe640 10 client.?.objecter ms_handle_connect 0x7f3ba413f710 2024-03-25T18:15:31.474 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.473+0000 7f3b8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:31.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.477+0000 7f3b8b7fe640 1 -- 172.21.15.156:0/905291311 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942677363 0 0) 0x7f3ba41d82f0 con 0x7f3ba40a98f0 2024-03-25T18:15:31.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.477+0000 7f3b8b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:31.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.477+0000 7f3b8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:31.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.477+0000 7f3b8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:31.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.477+0000 7f3b8b7fe640 10 monclient(hunting): my global_id is 4556 2024-03-25T18:15:31.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.477+0000 7f3b8b7fe640 1 -- 172.21.15.156:0/905291311 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ba4138b60 con 0x7f3ba40a98f0 2024-03-25T18:15:31.480 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.479+0000 7f3b8b7fe640 1 -- 172.21.15.156:0/905291311 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2046577580 0 0) 0x7f3ba4137e40 con 0x7f3ba413f710 2024-03-25T18:15:31.480 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.479+0000 7f3b8b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:31.480 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.479+0000 7f3b8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:31.480 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.479+0000 7f3b8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:31.480 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.479+0000 7f3b8b7fe640 10 monclient(hunting): my global_id is 4552 2024-03-25T18:15:31.480 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.479+0000 7f3b8b7fe640 1 -- 172.21.15.156:0/905291311 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ba41d82f0 con 0x7f3ba413f710 2024-03-25T18:15:31.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.480+0000 7f3b8b7fe640 1 -- 172.21.15.156:0/905291311 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4118821915 0 0) 0x7f3b90004eb0 con 0x7f3ba41430c0 2024-03-25T18:15:31.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.480+0000 7f3b8b7fe640 1 -- 172.21.15.156:0/905291311 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ba4137e40 con 0x7f3ba41430c0 2024-03-25T18:15:31.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.480+0000 7f3b8b7fe640 1 -- 172.21.15.156:0/905291311 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f3b9c001f40 con 0x7f3ba41430c0 2024-03-25T18:15:31.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.480+0000 7f3b8b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:31.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.480+0000 7f3b8b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:31.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.480+0000 7f3b8b7fe640 10 monclient(hunting): dump: 2024-03-25T18:15:31.482 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:31.482 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:31.482 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:31.482 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:31.482 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:31.482 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:31.482 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:31.482 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:31.482 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:31.482 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:31.482 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.480+0000 7f3b8b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:31.489 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.488+0000 7f3b8b7fe640 1 -- 172.21.15.156:0/905291311 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1996622969 0 0) 0x7f3ba41d82f0 con 0x7f3ba413f710 2024-03-25T18:15:31.489 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.488+0000 7f3b8b7fe640 1 -- 172.21.15.156:0/905291311 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b90004eb0 con 0x7f3ba413f710 2024-03-25T18:15:31.489 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.488+0000 7f3b8b7fe640 1 -- 172.21.15.156:0/905291311 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f3b98001860 con 0x7f3ba413f710 2024-03-25T18:15:31.489 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.488+0000 7f3b8b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:31.490 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.488+0000 7f3b8b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:31.490 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.488+0000 7f3b8b7fe640 10 monclient(hunting): dump: 2024-03-25T18:15:31.490 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:31.490 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:31.490 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:31.490 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:31.490 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:31.490 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:31.490 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:31.490 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:31.490 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:31.490 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:31.494 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.493+0000 7f3b8b7fe640 1 -- 172.21.15.156:0/905291311 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2381774632 0 0) 0x7f3ba4137e40 con 0x7f3ba41430c0 2024-03-25T18:15:31.495 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.493+0000 7f3b8b7fe640 1 -- 172.21.15.156:0/905291311 >> v1:172.21.15.156:6790/0 conn(0x7f3ba40a98f0 legacy=0x7f3ba4137730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:31.495 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.493+0000 7f3b8b7fe640 1 -- 172.21.15.156:0/905291311 >> v1:172.21.15.194:6789/0 conn(0x7f3ba413f710 legacy=0x7f3ba41d7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:31.495 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.493+0000 7f3b8b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:15:31.495 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.493+0000 7f3b8b7fe640 1 monclient: found mon.a 2024-03-25T18:15:31.495 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.493+0000 7f3b8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:31.495 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.493+0000 7f3b8b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:15:31.495 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.493+0000 7f3b8b7fe640 1 -- 172.21.15.156:0/905291311 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba4134e70 con 0x7f3ba41430c0 2024-03-25T18:15:31.495 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.493+0000 7f3b8b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:15:31.495 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.493+0000 7f3b8b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:15:31.495 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.493+0000 7f3b8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:31.495 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.493+0000 7f3ba9546640 5 monclient: authenticate success, global_id 4545 2024-03-25T18:15:31.495 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.493+0000 7f3ba9546640 10 monclient: _renew_subs 2024-03-25T18:15:31.495 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.493+0000 7f3ba9546640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:15:31.495 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.493+0000 7f3ba9546640 1 -- 172.21.15.156:0/905291311 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ba41350c0 con 0x7f3ba41430c0 2024-03-25T18:15:31.495 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.493+0000 7f3ba9546640 10 client.4545.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:31.495 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.493+0000 7f3ba9546640 10 monclient: _renew_subs 2024-03-25T18:15:31.496 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.493+0000 7f3ba9546640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:15:31.496 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.493+0000 7f3ba9546640 1 -- 172.21.15.156:0/905291311 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ba4134800 con 0x7f3ba41430c0 2024-03-25T18:15:31.496 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.493+0000 7f3ba9546640 1 librados: init done 2024-03-25T18:15:31.496 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.494+0000 7f3ba9546640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:31.496 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.494+0000 7f3ba9546640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:31.496 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.494+0000 7f3ba9546640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:15:31.497 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.494+0000 7f3ba9546640 1 -- 172.21.15.156:0/905291311 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ba414f330 con 0x7f3ba41430c0 2024-03-25T18:15:31.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.810+0000 7f3b8b7fe640 1 -- 172.21.15.156:0/905291311 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b9c004e10 con 0x7f3ba41430c0 2024-03-25T18:15:31.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.810+0000 7f3b8b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:31.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.810+0000 7f3b8b7fe640 1 -- 172.21.15.156:0/905291311 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f3b9c0015d0 con 0x7f3ba41430c0 2024-03-25T18:15:31.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.810+0000 7f3b8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:31.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.810+0000 7f3b8b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:31.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.810+0000 7f3b8b7fe640 10 monclient: dump: 2024-03-25T18:15:31.812 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:31.812 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:31.812 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:31.812 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:31.812 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:31.812 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:31.812 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:31.813 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:31.813 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:31.813 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:31.813 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.810+0000 7f3ba17fa640 4 set_mon_vals no callback set 2024-03-25T18:15:31.813 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.811+0000 7f3b8b7fe640 1 -- 172.21.15.156:0/905291311 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f3b9c01adf0 con 0x7f3ba41430c0 2024-03-25T18:15:31.813 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.812+0000 7f3ba2ffd640 10 monclient: get_auth_request con 0x7f3b9006df70 auth_method 0 2024-03-25T18:15:31.813 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.812+0000 7f3b8b7fe640 10 client.4545.objecter ms_handle_connect 0x7f3b9006df70 2024-03-25T18:15:31.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.819+0000 7f3b8b7fe640 1 -- 172.21.15.156:0/905291311 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f3b9c054a80 con 0x7f3ba41430c0 2024-03-25T18:15:31.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.819+0000 7f3b8b7fe640 10 client.4545.objecter ms_dispatch 0x7f3ba4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:31.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.819+0000 7f3b8b7fe640 3 client.4545.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:31.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.819+0000 7f3b8b7fe640 3 client.4545.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:31.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.819+0000 7f3b8b7fe640 20 client.4545.objecter dump_active .. 0 homeless 2024-03-25T18:15:31.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.819+0000 7f3b8b7fe640 1 -- 172.21.15.156:0/905291311 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3b9c0503c0 con 0x7f3ba41430c0 2024-03-25T18:15:31.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.819+0000 7f3b8b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:31.820 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:31.819+0000 7f3b8b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:32.073 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:32.072+0000 7f3ba9546640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:32.073 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:32.072+0000 7f3ba9546640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:32.073 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:32.072+0000 7f3ba9546640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:15:32.073 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:32.072+0000 7f3ba9546640 1 -- 172.21.15.156:0/905291311 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ba4074cc0 con 0x7f3ba41430c0 2024-03-25T18:15:32.100 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:32.099+0000 7f3b8b7fe640 1 -- 172.21.15.156:0/905291311 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f3b9c0524c0 con 0x7f3ba41430c0 2024-03-25T18:15:32.100 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:32.099+0000 7f3b8b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:32.100 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:32.099+0000 7f3b8b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:32.100 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:15:32.100 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:32.103 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:32.101+0000 7f3ba9546640 10 librados: watch_flush enter 2024-03-25T18:15:32.103 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:32.101+0000 7f3ba9546640 10 librados: watch_flush exit 2024-03-25T18:15:32.103 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:32.101+0000 7f3ba9546640 20 client.4545.objecter shutdown clearing up homeless session... 2024-03-25T18:15:32.103 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:32.101+0000 7f3ba9546640 10 client.4545.objecter successfully canceled tick 2024-03-25T18:15:32.103 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:32.101+0000 7f3ba9546640 1 -- 172.21.15.156:0/905291311 >> v1:172.21.15.194:6800/2946427713 conn(0x7f3b9006df70 legacy=0x7f3b90070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:32.103 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:32.101+0000 7f3ba9546640 10 monclient: shutdown 2024-03-25T18:15:32.103 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:32.101+0000 7f3ba9546640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:32.103 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:32.101+0000 7f3ba9546640 1 -- 172.21.15.156:0/905291311 >> v1:172.21.15.156:6789/0 conn(0x7f3ba41430c0 legacy=0x7f3ba4133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:32.103 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:32.101+0000 7f3ba9546640 1 -- 172.21.15.156:0/905291311 shutdown_connections 2024-03-25T18:15:32.103 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:32.101+0000 7f3ba9546640 1 -- 172.21.15.156:0/905291311 >> 172.21.15.156:0/905291311 conn(0x7f3ba40b7ea0 msgr2=0x7f3ba4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:32.103 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:32.101+0000 7f3ba9546640 1 -- 172.21.15.156:0/905291311 shutdown_connections 2024-03-25T18:15:32.104 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:32.101+0000 7f3ba9546640 1 -- 172.21.15.156:0/905291311 wait complete. 2024-03-25T18:15:32.104 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:32.101+0000 7f3ba9546640 1 librados: shutdown 2024-03-25T18:15:32.114 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:38.921 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:39.034 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.033+0000 7f57fb62a640 10 monclient: get_monmap_and_config 2024-03-25T18:15:39.035 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.033+0000 7f57fb62a640 10 monclient: build_initial_monmap 2024-03-25T18:15:39.035 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.033+0000 7f57fb62a640 10 monclient: monmap: 2024-03-25T18:15:39.035 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:15:39.035 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:39.035 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:15:39.034499+0000 2024-03-25T18:15:39.035 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:15:39.034499+0000 2024-03-25T18:15:39.035 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:39.035 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:39.035 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:15:39.035 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:15:39.035 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:15:39.035 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:39.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.034+0000 7f57fb62a640 1 Processor -- start 2024-03-25T18:15:39.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.034+0000 7f57fb62a640 1 -- start start 2024-03-25T18:15:39.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.034+0000 7f57fb62a640 10 monclient: init 2024-03-25T18:15:39.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.035+0000 7f57fb62a640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:39.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.035+0000 7f57fb62a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:39.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.035+0000 7f57fb62a640 10 monclient(hunting): picked mon.noname-b con 0x7f57f40a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:15:39.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.035+0000 7f57fb62a640 10 monclient(hunting): picked mon.noname-a con 0x7f57f413f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:15:39.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.035+0000 7f57fb62a640 10 monclient(hunting): picked mon.noname-c con 0x7f57f41430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:15:39.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.035+0000 7f57fb62a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f4142520 con 0x7f57f413f710 2024-03-25T18:15:39.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.035+0000 7f57fb62a640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f40b28a0 con 0x7f57f41430c0 2024-03-25T18:15:39.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.035+0000 7f57fb62a640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f40b2a30 con 0x7f57f40a98f0 2024-03-25T18:15:39.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.035+0000 7f57fb62a640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:39.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.035+0000 7f57fb62a640 10 monclient(hunting): authenticate will time out at 7420.346022s 2024-03-25T18:15:39.047 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.045+0000 7f57f93c6640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f57f40a98f0 0x7f57f40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:39444/0 (socket says 172.21.15.156:39444) 2024-03-25T18:15:39.047 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.046+0000 7f57f93c6640 1 -- 172.21.15.156:0/481803289 learned_addr learned my addr 172.21.15.156:0/481803289 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:15:39.062 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.061+0000 7f57ebfff640 1 -- 172.21.15.156:0/481803289 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3077428710 0 0) 0x7f57f40b2a30 con 0x7f57f40a98f0 2024-03-25T18:15:39.062 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.061+0000 7f57ebfff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:39.062 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.061+0000 7f57ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:39.062 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.061+0000 7f57ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:39.062 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.061+0000 7f57ebfff640 10 monclient(hunting): my global_id is 4559 2024-03-25T18:15:39.062 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.061+0000 7f57ebfff640 1 -- 172.21.15.156:0/481803289 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57e0003520 con 0x7f57f40a98f0 2024-03-25T18:15:39.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.070+0000 7f57ebfff640 1 -- 172.21.15.156:0/481803289 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2361707112 0 0) 0x7f57f40b28a0 con 0x7f57f41430c0 2024-03-25T18:15:39.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.070+0000 7f57ebfff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:39.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.070+0000 7f57ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:39.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.070+0000 7f57ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:39.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.070+0000 7f57ebfff640 10 monclient(hunting): my global_id is 4555 2024-03-25T18:15:39.071 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.070+0000 7f57ebfff640 1 -- 172.21.15.156:0/481803289 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57f40b2a30 con 0x7f57f41430c0 2024-03-25T18:15:39.075 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.074+0000 7f57ebfff640 1 -- 172.21.15.156:0/481803289 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4253264707 0 0) 0x7f57e0003520 con 0x7f57f40a98f0 2024-03-25T18:15:39.075 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.074+0000 7f57ebfff640 1 -- 172.21.15.156:0/481803289 >> v1:172.21.15.194:6789/0 conn(0x7f57f41430c0 legacy=0x7f57f414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:39.075 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.074+0000 7f57ebfff640 1 -- 172.21.15.156:0/481803289 >> v1:172.21.15.156:6789/0 conn(0x7f57f413f710 legacy=0x7f57f41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:39.075 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.074+0000 7f57ebfff640 10 monclient: _finish_hunting 0 2024-03-25T18:15:39.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.074+0000 7f57ebfff640 1 monclient: found mon.noname-b 2024-03-25T18:15:39.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.074+0000 7f57ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:39.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.074+0000 7f57ebfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.156:6790/0 2024-03-25T18:15:39.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.074+0000 7f57ebfff640 1 -- 172.21.15.156:0/481803289 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f40b44c0 con 0x7f57f40a98f0 2024-03-25T18:15:39.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.074+0000 7f57ebfff640 10 monclient: _finish_auth 0 2024-03-25T18:15:39.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.074+0000 7f57ebfff640 10 monclient: _check_auth_tickets 2024-03-25T18:15:39.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.074+0000 7f57ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:39.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.074+0000 7f57ebfff640 1 -- 172.21.15.156:0/481803289 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f57f0001690 con 0x7f57f40a98f0 2024-03-25T18:15:39.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.074+0000 7f57ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:39.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.074+0000 7f57ebfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:39.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.074+0000 7f57ebfff640 10 monclient: dump: 2024-03-25T18:15:39.076 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:39.076 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:39.076 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:39.076 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:39.077 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:39.077 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:39.077 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:39.077 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:39.077 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:39.077 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:39.077 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.074+0000 7f57fb62a640 5 monclient: authenticate success, global_id 4559 2024-03-25T18:15:39.077 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.074+0000 7f57fb62a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:39.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.079+0000 7f57ebfff640 1 -- 172.21.15.156:0/481803289 <== mon.2 v1:172.21.15.156:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57f00017f0 con 0x7f57f40a98f0 2024-03-25T18:15:39.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.079+0000 7f57ebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:39.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.079+0000 7f57ebfff640 1 -- 172.21.15.156:0/481803289 <== mon.2 v1:172.21.15.156:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f57f0001c00 con 0x7f57f40a98f0 2024-03-25T18:15:39.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.079+0000 7f57fb62a640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:39.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.079+0000 7f57fb62a640 4 set_mon_vals no callback set 2024-03-25T18:15:39.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.079+0000 7f57ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:39.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.079+0000 7f57ebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:39.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.079+0000 7f57ebfff640 10 monclient: dump: 2024-03-25T18:15:39.080 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:39.080 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:39.080 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:39.080 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:39.081 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:39.081 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:39.081 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:39.081 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:39.081 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:39.081 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:39.082 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.079+0000 7f57fb62a640 10 monclient: shutdown 2024-03-25T18:15:39.082 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.079+0000 7f57fb62a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:39.083 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.079+0000 7f57fb62a640 1 -- 172.21.15.156:0/481803289 >> v1:172.21.15.156:6790/0 conn(0x7f57f40a98f0 legacy=0x7f57f40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:39.083 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.080+0000 7f57fb62a640 1 -- 172.21.15.156:0/481803289 shutdown_connections 2024-03-25T18:15:39.083 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.080+0000 7f57fb62a640 1 -- 172.21.15.156:0/481803289 >> 172.21.15.156:0/481803289 conn(0x7f57f40b7ea0 msgr2=0x7f57f40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:39.083 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.080+0000 7f57fb62a640 1 -- 172.21.15.156:0/481803289 shutdown_connections 2024-03-25T18:15:39.083 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.080+0000 7f57fb62a640 1 -- 172.21.15.156:0/481803289 wait complete. 2024-03-25T18:15:39.083 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.080+0000 7f57fb62a640 10 monclient: build_initial_monmap 2024-03-25T18:15:39.083 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.080+0000 7f57fb62a640 10 monclient: monmap: 2024-03-25T18:15:39.083 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:15:39.083 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:39.083 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:15:39.083 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:15:39.083 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:39.083 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:15:39.083 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:15:39.083 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:15:39.084 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:15:39.084 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:39.084 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.081+0000 7f57fb62a640 1 librados: starting msgr at 2024-03-25T18:15:39.084 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.081+0000 7f57fb62a640 1 librados: starting objecter 2024-03-25T18:15:39.084 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.081+0000 7f57fb62a640 1 Processor -- start 2024-03-25T18:15:39.084 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.081+0000 7f57fb62a640 1 -- start start 2024-03-25T18:15:39.084 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.081+0000 7f57fb62a640 1 librados: setting wanted keys 2024-03-25T18:15:39.084 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.081+0000 7f57fb62a640 1 librados: calling monclient init 2024-03-25T18:15:39.084 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.081+0000 7f57fb62a640 10 monclient: init 2024-03-25T18:15:39.084 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.081+0000 7f57fb62a640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:39.084 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.081+0000 7f57fb62a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:15:39.084 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.081+0000 7f57fb62a640 10 monclient(hunting): picked mon.noname-c con 0x7f57f40a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:15:39.084 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.081+0000 7f57fb62a640 10 monclient(hunting): picked mon.noname-b con 0x7f57f413f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:15:39.084 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.081+0000 7f57fb62a640 10 monclient(hunting): picked mon.noname-a con 0x7f57f41430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:15:39.084 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.081+0000 7f57fb62a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f40ab9e0 con 0x7f57f41430c0 2024-03-25T18:15:39.084 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.081+0000 7f57fb62a640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f4138c20 con 0x7f57f413f710 2024-03-25T18:15:39.084 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.081+0000 7f57fb62a640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f4138f00 con 0x7f57f40a98f0 2024-03-25T18:15:39.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.081+0000 7f57fb62a640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:39.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.081+0000 7f57fb62a640 10 monclient(hunting): authenticate will time out at 7420.392744s 2024-03-25T18:15:39.092 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.091+0000 7f57f93c6640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f57f40a98f0 0x7f57f4137fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:39472/0 (socket says 172.21.15.156:39472) 2024-03-25T18:15:39.092 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.091+0000 7f57f93c6640 1 -- 172.21.15.156:0/1425642219 learned_addr learned my addr 172.21.15.156:0/1425642219 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:15:39.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.092+0000 7f57e9ffb640 10 client.?.objecter ms_handle_connect 0x7f57f40a98f0 2024-03-25T18:15:39.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.092+0000 7f57e9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:39.096 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.095+0000 7f57e9ffb640 10 client.?.objecter ms_handle_connect 0x7f57f41430c0 2024-03-25T18:15:39.096 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.095+0000 7f57e9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:39.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.098+0000 7f57e9ffb640 10 client.?.objecter ms_handle_connect 0x7f57f413f710 2024-03-25T18:15:39.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.098+0000 7f57e9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:39.099 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.098+0000 7f57e9ffb640 1 -- 172.21.15.156:0/1425642219 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 219245844 0 0) 0x7f57f4138f00 con 0x7f57f40a98f0 2024-03-25T18:15:39.099 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.098+0000 7f57e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:39.100 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.098+0000 7f57e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:39.100 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.098+0000 7f57e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:39.100 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.098+0000 7f57e9ffb640 10 monclient(hunting): my global_id is 4562 2024-03-25T18:15:39.100 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.098+0000 7f57e9ffb640 1 -- 172.21.15.156:0/1425642219 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57d8004eb0 con 0x7f57f40a98f0 2024-03-25T18:15:39.113 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.112+0000 7f57e9ffb640 1 -- 172.21.15.156:0/1425642219 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3624542868 0 0) 0x7f57d8004eb0 con 0x7f57f40a98f0 2024-03-25T18:15:39.113 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.112+0000 7f57e9ffb640 1 -- 172.21.15.156:0/1425642219 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57f4138f00 con 0x7f57f40a98f0 2024-03-25T18:15:39.113 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.112+0000 7f57e9ffb640 1 -- 172.21.15.156:0/1425642219 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f57f0001bc0 con 0x7f57f40a98f0 2024-03-25T18:15:39.113 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.112+0000 7f57e9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:39.113 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.112+0000 7f57e9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:39.113 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.112+0000 7f57e9ffb640 10 monclient(hunting): dump: 2024-03-25T18:15:39.114 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:39.114 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:39.114 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:39.114 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:39.114 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:39.114 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:39.114 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:39.114 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:39.114 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:39.114 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:39.114 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.112+0000 7f57e9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:39.118 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.117+0000 7f57e9ffb640 1 -- 172.21.15.156:0/1425642219 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439752185 0 0) 0x7f57f40ab9e0 con 0x7f57f41430c0 2024-03-25T18:15:39.118 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.117+0000 7f57e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:39.118 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.117+0000 7f57e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:39.118 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.117+0000 7f57e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:39.118 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.117+0000 7f57e9ffb640 10 monclient(hunting): my global_id is 4551 2024-03-25T18:15:39.118 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.117+0000 7f57e9ffb640 1 -- 172.21.15.156:0/1425642219 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57d8004eb0 con 0x7f57f41430c0 2024-03-25T18:15:39.124 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.123+0000 7f57e9ffb640 1 -- 172.21.15.156:0/1425642219 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360576747 0 0) 0x7f57f4138c20 con 0x7f57f413f710 2024-03-25T18:15:39.124 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.123+0000 7f57e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:39.124 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.123+0000 7f57e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:39.124 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.123+0000 7f57e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:39.124 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.123+0000 7f57e9ffb640 10 monclient(hunting): my global_id is 4558 2024-03-25T18:15:39.124 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.123+0000 7f57e9ffb640 1 -- 172.21.15.156:0/1425642219 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57f40ab9e0 con 0x7f57f413f710 2024-03-25T18:15:39.131 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.130+0000 7f57e9ffb640 1 -- 172.21.15.156:0/1425642219 <== mon.2 v1:172.21.15.156:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3136840452 0 0) 0x7f57f4138f00 con 0x7f57f40a98f0 2024-03-25T18:15:39.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.130+0000 7f57e9ffb640 1 -- 172.21.15.156:0/1425642219 >> v1:172.21.15.194:6789/0 conn(0x7f57f413f710 legacy=0x7f57f41d7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:39.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.130+0000 7f57e9ffb640 1 -- 172.21.15.156:0/1425642219 >> v1:172.21.15.156:6789/0 conn(0x7f57f41430c0 legacy=0x7f57f41e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:39.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.130+0000 7f57e9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:15:39.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.130+0000 7f57e9ffb640 1 monclient: found mon.c 2024-03-25T18:15:39.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.130+0000 7f57e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:39.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.130+0000 7f57e9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:15:39.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.130+0000 7f57e9ffb640 1 -- 172.21.15.156:0/1425642219 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f414e7a0 con 0x7f57f40a98f0 2024-03-25T18:15:39.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.130+0000 7f57e9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:15:39.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.130+0000 7f57e9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:15:39.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.130+0000 7f57e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:39.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.130+0000 7f57fb62a640 5 monclient: authenticate success, global_id 4562 2024-03-25T18:15:39.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.130+0000 7f57fb62a640 10 monclient: _renew_subs 2024-03-25T18:15:39.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.130+0000 7f57fb62a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:15:39.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.130+0000 7f57fb62a640 1 -- 172.21.15.156:0/1425642219 --> v1:172.21.15.156:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57f41386d0 con 0x7f57f40a98f0 2024-03-25T18:15:39.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.130+0000 7f57fb62a640 10 client.4562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:39.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.130+0000 7f57fb62a640 10 monclient: _renew_subs 2024-03-25T18:15:39.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.130+0000 7f57fb62a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:15:39.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.130+0000 7f57fb62a640 1 -- 172.21.15.156:0/1425642219 --> v1:172.21.15.156:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f57f41d8060 con 0x7f57f40a98f0 2024-03-25T18:15:39.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.130+0000 7f57fb62a640 1 librados: init done 2024-03-25T18:15:39.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.131+0000 7f57fb62a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:39.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.131+0000 7f57fb62a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:39.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.131+0000 7f57fb62a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:15:39.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.131+0000 7f57fb62a640 1 -- 172.21.15.156:0/1425642219 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57f40627b0 con 0x7f57f40a98f0 2024-03-25T18:15:39.441 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.440+0000 7f57e9ffb640 1 -- 172.21.15.156:0/1425642219 <== mon.2 v1:172.21.15.156:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57f00011f0 con 0x7f57f40a98f0 2024-03-25T18:15:39.441 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.440+0000 7f57e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:39.442 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.440+0000 7f57e9ffb640 1 -- 172.21.15.156:0/1425642219 <== mon.2 v1:172.21.15.156:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f57f0004c70 con 0x7f57f40a98f0 2024-03-25T18:15:39.442 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.440+0000 7f57e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:39.442 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.440+0000 7f57e9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:39.442 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.440+0000 7f57e9ffb640 10 monclient: dump: 2024-03-25T18:15:39.442 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:39.442 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:39.442 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:39.442 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:39.442 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:39.442 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:39.442 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:39.442 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:39.443 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:39.443 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:39.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.440+0000 7f57eb7fe640 4 set_mon_vals no callback set 2024-03-25T18:15:39.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.441+0000 7f57e9ffb640 1 -- 172.21.15.156:0/1425642219 <== mon.2 v1:172.21.15.156:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f57f0004e90 con 0x7f57f40a98f0 2024-03-25T18:15:39.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.442+0000 7f57f8bc5640 10 monclient: get_auth_request con 0x7f57d806de10 auth_method 0 2024-03-25T18:15:39.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.442+0000 7f57e9ffb640 10 client.4562.objecter ms_handle_connect 0x7f57d806de10 2024-03-25T18:15:39.448 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.447+0000 7f57e9ffb640 1 -- 172.21.15.156:0/1425642219 <== mon.2 v1:172.21.15.156:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f57f00549a0 con 0x7f57f40a98f0 2024-03-25T18:15:39.448 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.447+0000 7f57e9ffb640 10 client.4562.objecter ms_dispatch 0x7f57f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:39.448 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.447+0000 7f57e9ffb640 3 client.4562.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:39.448 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.447+0000 7f57e9ffb640 3 client.4562.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:39.448 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.447+0000 7f57e9ffb640 20 client.4562.objecter dump_active .. 0 homeless 2024-03-25T18:15:39.448 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.447+0000 7f57e9ffb640 1 -- 172.21.15.156:0/1425642219 <== mon.2 v1:172.21.15.156:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f57f00502e0 con 0x7f57f40a98f0 2024-03-25T18:15:39.448 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.447+0000 7f57e9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:39.448 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.447+0000 7f57e9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:39.570 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:07:45.560 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:39.574 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:07:45.565 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:39.618 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:07:45.608 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:39.631 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:07:45.597 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:39.636 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:07:45.603 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:39.644 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:07:45.634 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:39.654 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:07:45.644 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:39.656 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.655+0000 7f57fb62a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:39.656 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.655+0000 7f57fb62a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:39.656 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.655+0000 7f57fb62a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:15:39.656 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.655+0000 7f57fb62a640 1 -- 172.21.15.156:0/1425642219 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57f40008d0 con 0x7f57f40a98f0 2024-03-25T18:15:39.663 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:07:45.630 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:39.706 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.705+0000 7f57e9ffb640 1 -- 172.21.15.156:0/1425642219 <== mon.2 v1:172.21.15.156:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f57f00523e0 con 0x7f57f40a98f0 2024-03-25T18:15:39.707 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.705+0000 7f57e9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:39.707 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.705+0000 7f57e9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:39.707 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:15:39.707 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:39.709 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.708+0000 7f57fb62a640 10 librados: watch_flush enter 2024-03-25T18:15:39.709 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.708+0000 7f57fb62a640 10 librados: watch_flush exit 2024-03-25T18:15:39.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.708+0000 7f57fb62a640 20 client.4562.objecter shutdown clearing up homeless session... 2024-03-25T18:15:39.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.708+0000 7f57fb62a640 10 client.4562.objecter successfully canceled tick 2024-03-25T18:15:39.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.708+0000 7f57fb62a640 1 -- 172.21.15.156:0/1425642219 >> v1:172.21.15.194:6800/2946427713 conn(0x7f57d806de10 legacy=0x7f57d80702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:39.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.708+0000 7f57fb62a640 10 monclient: shutdown 2024-03-25T18:15:39.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.708+0000 7f57fb62a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:39.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.708+0000 7f57fb62a640 1 -- 172.21.15.156:0/1425642219 >> v1:172.21.15.156:6790/0 conn(0x7f57f40a98f0 legacy=0x7f57f4137fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:39.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.708+0000 7f57fb62a640 1 -- 172.21.15.156:0/1425642219 shutdown_connections 2024-03-25T18:15:39.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.708+0000 7f57fb62a640 1 -- 172.21.15.156:0/1425642219 >> 172.21.15.156:0/1425642219 conn(0x7f57f40b7ea0 msgr2=0x7f57f4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:39.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.708+0000 7f57fb62a640 1 -- 172.21.15.156:0/1425642219 shutdown_connections 2024-03-25T18:15:39.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.708+0000 7f57fb62a640 1 -- 172.21.15.156:0/1425642219 wait complete. 2024-03-25T18:15:39.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:39.708+0000 7f57fb62a640 1 librados: shutdown 2024-03-25T18:15:39.711 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:07:45.700 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:39.720 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:46.527 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:46.644 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.643+0000 7f77bac1e640 10 monclient: get_monmap_and_config 2024-03-25T18:15:46.645 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.643+0000 7f77bac1e640 10 monclient: build_initial_monmap 2024-03-25T18:15:46.645 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.643+0000 7f77bac1e640 10 monclient: monmap: 2024-03-25T18:15:46.645 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:15:46.645 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:46.645 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:15:46.644854+0000 2024-03-25T18:15:46.645 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:15:46.644854+0000 2024-03-25T18:15:46.645 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:46.645 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:46.646 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:15:46.646 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:15:46.646 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:15:46.646 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:46.647 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.644+0000 7f77bac1e640 1 Processor -- start 2024-03-25T18:15:46.647 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.644+0000 7f77bac1e640 1 -- start start 2024-03-25T18:15:46.647 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.644+0000 7f77bac1e640 10 monclient: init 2024-03-25T18:15:46.647 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.645+0000 7f77bac1e640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:46.647 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.645+0000 7f77bac1e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:46.647 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.645+0000 7f77bac1e640 10 monclient(hunting): picked mon.noname-a con 0x7f77b40b4d60 addr v1:172.21.15.156:6789/0 2024-03-25T18:15:46.648 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.645+0000 7f77bac1e640 10 monclient(hunting): picked mon.noname-c con 0x7f77b40b1730 addr v1:172.21.15.194:6789/0 2024-03-25T18:15:46.648 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.645+0000 7f77bac1e640 10 monclient(hunting): picked mon.noname-b con 0x7f77b414be90 addr v1:172.21.15.156:6790/0 2024-03-25T18:15:46.648 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.645+0000 7f77bac1e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b40af940 con 0x7f77b40b4d60 2024-03-25T18:15:46.648 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.645+0000 7f77bac1e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b40b0a70 con 0x7f77b40b1730 2024-03-25T18:15:46.648 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.645+0000 7f77bac1e640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b40b0d50 con 0x7f77b414be90 2024-03-25T18:15:46.648 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.645+0000 7f77bac1e640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:46.648 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.645+0000 7f77bac1e640 10 monclient(hunting): authenticate will time out at 7427.956418s 2024-03-25T18:15:46.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.652+0000 7f77b3fff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f77b40b1730 0x7f77b40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:53202/0 (socket says 172.21.15.156:53202) 2024-03-25T18:15:46.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.652+0000 7f77b3fff640 1 -- 172.21.15.156:0/4168657345 learned_addr learned my addr 172.21.15.156:0/4168657345 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:15:46.671 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.670+0000 7f77b37fe640 1 -- 172.21.15.156:0/4168657345 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231818586 0 0) 0x7f77b40b0a70 con 0x7f77b40b1730 2024-03-25T18:15:46.671 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.670+0000 7f77b37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:46.671 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.670+0000 7f77b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:46.671 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.670+0000 7f77b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:46.671 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.670+0000 7f77b37fe640 10 monclient(hunting): my global_id is 4561 2024-03-25T18:15:46.672 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.670+0000 7f77b37fe640 1 -- 172.21.15.156:0/4168657345 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77a0003520 con 0x7f77b40b1730 2024-03-25T18:15:46.672 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.670+0000 7f77b37fe640 1 -- 172.21.15.156:0/4168657345 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053831200 0 0) 0x7f77b40b0d50 con 0x7f77b414be90 2024-03-25T18:15:46.672 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.670+0000 7f77b37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:46.672 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.670+0000 7f77b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:46.672 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.670+0000 7f77b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:46.672 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.670+0000 7f77b37fe640 10 monclient(hunting): my global_id is 4565 2024-03-25T18:15:46.672 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.670+0000 7f77b37fe640 1 -- 172.21.15.156:0/4168657345 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77b40b0a70 con 0x7f77b414be90 2024-03-25T18:15:46.685 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.683+0000 7f77b37fe640 1 -- 172.21.15.156:0/4168657345 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3786945186 0 0) 0x7f77b40b0a70 con 0x7f77b414be90 2024-03-25T18:15:46.685 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.683+0000 7f77b37fe640 1 -- 172.21.15.156:0/4168657345 >> v1:172.21.15.194:6789/0 conn(0x7f77b40b1730 legacy=0x7f77b40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:46.685 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.684+0000 7f77b37fe640 1 -- 172.21.15.156:0/4168657345 >> v1:172.21.15.156:6789/0 conn(0x7f77b40b4d60 legacy=0x7f77b413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:46.685 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.684+0000 7f77b37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:15:46.685 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.684+0000 7f77b37fe640 1 monclient: found mon.noname-b 2024-03-25T18:15:46.685 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.684+0000 7f77b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:46.685 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.684+0000 7f77b37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.156:6790/0 2024-03-25T18:15:46.685 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.684+0000 7f77b37fe640 1 -- 172.21.15.156:0/4168657345 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b40b1030 con 0x7f77b414be90 2024-03-25T18:15:46.685 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.684+0000 7f77b37fe640 10 monclient: _finish_auth 0 2024-03-25T18:15:46.685 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.684+0000 7f77b37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:15:46.685 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.684+0000 7f77b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:46.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.684+0000 7f77b37fe640 1 -- 172.21.15.156:0/4168657345 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f77a4001670 con 0x7f77b414be90 2024-03-25T18:15:46.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.684+0000 7f77bac1e640 5 monclient: authenticate success, global_id 4565 2024-03-25T18:15:46.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.684+0000 7f77bac1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:46.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.684+0000 7f77b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:46.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.684+0000 7f77b37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:46.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.684+0000 7f77b37fe640 10 monclient: dump: 2024-03-25T18:15:46.686 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:46.686 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:46.686 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:46.686 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:46.686 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:46.686 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:46.686 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:46.686 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:46.687 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:46.687 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:46.687 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.684+0000 7f77bac1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:46.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.693+0000 7f77b37fe640 1 -- 172.21.15.156:0/4168657345 <== mon.2 v1:172.21.15.156:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77a40017d0 con 0x7f77b414be90 2024-03-25T18:15:46.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.693+0000 7f77b37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:46.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.693+0000 7f77b37fe640 1 -- 172.21.15.156:0/4168657345 <== mon.2 v1:172.21.15.156:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f77a4001c20 con 0x7f77b414be90 2024-03-25T18:15:46.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.693+0000 7f77bac1e640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:46.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.693+0000 7f77bac1e640 4 set_mon_vals no callback set 2024-03-25T18:15:46.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.693+0000 7f77b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:46.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.693+0000 7f77b37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:46.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.693+0000 7f77b37fe640 10 monclient: dump: 2024-03-25T18:15:46.695 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:46.695 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:46.695 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:46.695 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:46.695 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:46.695 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:46.695 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:46.695 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:46.695 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:46.695 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:46.696 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.694+0000 7f77bac1e640 10 monclient: shutdown 2024-03-25T18:15:46.696 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.694+0000 7f77bac1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:46.696 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.694+0000 7f77bac1e640 1 -- 172.21.15.156:0/4168657345 >> v1:172.21.15.156:6790/0 conn(0x7f77b414be90 legacy=0x7f77b414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:46.696 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.694+0000 7f77bac1e640 1 -- 172.21.15.156:0/4168657345 shutdown_connections 2024-03-25T18:15:46.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.694+0000 7f77bac1e640 1 -- 172.21.15.156:0/4168657345 >> 172.21.15.156:0/4168657345 conn(0x7f77b40617a0 msgr2=0x7f77b4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:46.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.694+0000 7f77bac1e640 1 -- 172.21.15.156:0/4168657345 shutdown_connections 2024-03-25T18:15:46.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.694+0000 7f77bac1e640 1 -- 172.21.15.156:0/4168657345 wait complete. 2024-03-25T18:15:46.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.695+0000 7f77bac1e640 10 monclient: build_initial_monmap 2024-03-25T18:15:46.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.695+0000 7f77bac1e640 10 monclient: monmap: 2024-03-25T18:15:46.697 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:15:46.697 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:46.697 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:15:46.697 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:15:46.697 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:46.697 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:15:46.697 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:15:46.697 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:15:46.697 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:15:46.697 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:46.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.695+0000 7f77bac1e640 1 librados: starting msgr at 2024-03-25T18:15:46.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.695+0000 7f77bac1e640 1 librados: starting objecter 2024-03-25T18:15:46.698 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.695+0000 7f77bac1e640 1 Processor -- start 2024-03-25T18:15:46.698 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.695+0000 7f77bac1e640 1 -- start start 2024-03-25T18:15:46.698 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.695+0000 7f77bac1e640 1 librados: setting wanted keys 2024-03-25T18:15:46.698 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.695+0000 7f77bac1e640 1 librados: calling monclient init 2024-03-25T18:15:46.698 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.695+0000 7f77bac1e640 10 monclient: init 2024-03-25T18:15:46.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.696+0000 7f77bac1e640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:46.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.696+0000 7f77bac1e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:15:46.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.696+0000 7f77bac1e640 10 monclient(hunting): picked mon.noname-c con 0x7f77b40b1730 addr v1:172.21.15.156:6790/0 2024-03-25T18:15:46.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.696+0000 7f77bac1e640 10 monclient(hunting): picked mon.noname-b con 0x7f77b40b4d60 addr v1:172.21.15.194:6789/0 2024-03-25T18:15:46.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.696+0000 7f77bac1e640 10 monclient(hunting): picked mon.noname-a con 0x7f77b414be90 addr v1:172.21.15.156:6789/0 2024-03-25T18:15:46.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.696+0000 7f77bac1e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b41d7520 con 0x7f77b414be90 2024-03-25T18:15:46.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.697+0000 7f77bac1e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b41d83a0 con 0x7f77b40b4d60 2024-03-25T18:15:46.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.697+0000 7f77bac1e640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b4142d30 con 0x7f77b40b1730 2024-03-25T18:15:46.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.697+0000 7f77bac1e640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:46.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.697+0000 7f77bac1e640 10 monclient(hunting): authenticate will time out at 7428.007891s 2024-03-25T18:15:46.704 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.703+0000 7f77b3fff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f77b40b4d60 0x7f77b4132db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:53262/0 (socket says 172.21.15.156:53262) 2024-03-25T18:15:46.704 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.703+0000 7f77b3fff640 1 -- 172.21.15.156:0/4199054136 learned_addr learned my addr 172.21.15.156:0/4199054136 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:15:46.707 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.706+0000 7f77b17fa640 10 client.?.objecter ms_handle_connect 0x7f77b40b4d60 2024-03-25T18:15:46.707 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.706+0000 7f77b17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:46.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.715+0000 7f77b17fa640 10 client.?.objecter ms_handle_connect 0x7f77b414be90 2024-03-25T18:15:46.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.715+0000 7f77b17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:46.719 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.718+0000 7f77b17fa640 1 -- 172.21.15.156:0/4199054136 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1237262378 0 0) 0x7f77b41d83a0 con 0x7f77b40b4d60 2024-03-25T18:15:46.719 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.718+0000 7f77b17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:46.720 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.718+0000 7f77b17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:46.720 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.718+0000 7f77b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:46.720 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.718+0000 7f77b17fa640 10 monclient(hunting): my global_id is 4564 2024-03-25T18:15:46.720 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.718+0000 7f77b17fa640 1 -- 172.21.15.156:0/4199054136 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7794004eb0 con 0x7f77b40b4d60 2024-03-25T18:15:46.723 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.722+0000 7f77b17fa640 10 client.?.objecter ms_handle_connect 0x7f77b40b1730 2024-03-25T18:15:46.723 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.722+0000 7f77b17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:46.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.727+0000 7f77b17fa640 1 -- 172.21.15.156:0/4199054136 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675997478 0 0) 0x7f77b41d7520 con 0x7f77b414be90 2024-03-25T18:15:46.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.727+0000 7f77b17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:46.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.727+0000 7f77b17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:46.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.727+0000 7f77b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:46.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.727+0000 7f77b17fa640 10 monclient(hunting): my global_id is 4557 2024-03-25T18:15:46.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.727+0000 7f77b17fa640 1 -- 172.21.15.156:0/4199054136 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77b41d83a0 con 0x7f77b414be90 2024-03-25T18:15:46.737 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.736+0000 7f77b17fa640 1 -- 172.21.15.156:0/4199054136 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1780714771 0 0) 0x7f7794004eb0 con 0x7f77b40b4d60 2024-03-25T18:15:46.737 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.736+0000 7f77b17fa640 1 -- 172.21.15.156:0/4199054136 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77b41d7520 con 0x7f77b40b4d60 2024-03-25T18:15:46.737 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.736+0000 7f77b17fa640 1 -- 172.21.15.156:0/4199054136 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f779c001990 con 0x7f77b40b4d60 2024-03-25T18:15:46.737 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.736+0000 7f77b17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:46.738 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.736+0000 7f77b17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:46.738 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.736+0000 7f77b17fa640 10 monclient(hunting): dump: 2024-03-25T18:15:46.738 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:46.738 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:46.738 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:46.738 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:46.738 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:46.738 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:46.738 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:46.738 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:46.738 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:46.738 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:46.738 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.736+0000 7f77b17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:46.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.742+0000 7f77b17fa640 1 -- 172.21.15.156:0/4199054136 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3539067158 0 0) 0x7f77b41d7520 con 0x7f77b40b4d60 2024-03-25T18:15:46.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.742+0000 7f77b17fa640 1 -- 172.21.15.156:0/4199054136 >> v1:172.21.15.156:6790/0 conn(0x7f77b40b1730 legacy=0x7f77b41d7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:46.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.742+0000 7f77b17fa640 1 -- 172.21.15.156:0/4199054136 >> v1:172.21.15.156:6789/0 conn(0x7f77b414be90 legacy=0x7f77b4141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:46.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.742+0000 7f77b17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:15:46.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.742+0000 7f77b17fa640 1 monclient: found mon.b 2024-03-25T18:15:46.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.742+0000 7f77b17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:46.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.742+0000 7f77b17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:15:46.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.742+0000 7f77b17fa640 1 -- 172.21.15.156:0/4199054136 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b41d7770 con 0x7f77b40b4d60 2024-03-25T18:15:46.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.742+0000 7f77b17fa640 10 monclient: _finish_auth 0 2024-03-25T18:15:46.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.742+0000 7f77b17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:15:46.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.742+0000 7f77b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:46.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.742+0000 7f77bac1e640 5 monclient: authenticate success, global_id 4564 2024-03-25T18:15:46.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.742+0000 7f77bac1e640 10 monclient: _renew_subs 2024-03-25T18:15:46.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.742+0000 7f77bac1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:15:46.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.742+0000 7f77bac1e640 1 -- 172.21.15.156:0/4199054136 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77b41334c0 con 0x7f77b40b4d60 2024-03-25T18:15:46.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.742+0000 7f77bac1e640 10 client.4564.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:46.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.742+0000 7f77bac1e640 10 monclient: _renew_subs 2024-03-25T18:15:46.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.742+0000 7f77bac1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:15:46.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.742+0000 7f77bac1e640 1 -- 172.21.15.156:0/4199054136 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77b413d0f0 con 0x7f77b40b4d60 2024-03-25T18:15:46.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.742+0000 7f77bac1e640 1 librados: init done 2024-03-25T18:15:46.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.743+0000 7f77bac1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:46.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.743+0000 7f77bac1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:46.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.743+0000 7f77bac1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:15:46.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:46.743+0000 7f77bac1e640 1 -- 172.21.15.156:0/4199054136 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77b40627b0 con 0x7f77b40b4d60 2024-03-25T18:15:47.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.013+0000 7f77b17fa640 1 -- 172.21.15.156:0/4199054136 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f779c0047e0 con 0x7f77b40b4d60 2024-03-25T18:15:47.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.013+0000 7f77b17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:47.015 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.013+0000 7f77b17fa640 1 -- 172.21.15.156:0/4199054136 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f779c004bb0 con 0x7f77b40b4d60 2024-03-25T18:15:47.015 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.013+0000 7f77b17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:47.015 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.013+0000 7f77b17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:47.015 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.013+0000 7f77b2ffd640 4 set_mon_vals no callback set 2024-03-25T18:15:47.015 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.013+0000 7f77b17fa640 10 monclient: dump: 2024-03-25T18:15:47.015 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:47.015 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:47.015 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:47.015 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:47.015 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:47.015 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:47.015 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:47.015 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:47.015 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:47.016 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:47.016 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.014+0000 7f77b17fa640 1 -- 172.21.15.156:0/4199054136 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f779c004dd0 con 0x7f77b40b4d60 2024-03-25T18:15:47.016 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.015+0000 7f77b89ba640 10 monclient: get_auth_request con 0x7f779406dce0 auth_method 0 2024-03-25T18:15:47.016 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.015+0000 7f77b17fa640 10 client.4564.objecter ms_handle_connect 0x7f779406dce0 2024-03-25T18:15:47.022 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.021+0000 7f77b17fa640 1 -- 172.21.15.156:0/4199054136 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f779c0548a0 con 0x7f77b40b4d60 2024-03-25T18:15:47.022 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.021+0000 7f77b17fa640 10 client.4564.objecter ms_dispatch 0x7f77b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:47.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.021+0000 7f77b17fa640 3 client.4564.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:47.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.021+0000 7f77b17fa640 3 client.4564.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:47.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.021+0000 7f77b17fa640 20 client.4564.objecter dump_active .. 0 homeless 2024-03-25T18:15:47.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.021+0000 7f77b17fa640 1 -- 172.21.15.156:0/4199054136 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f779c0503b0 con 0x7f77b40b4d60 2024-03-25T18:15:47.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.021+0000 7f77b17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:47.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.021+0000 7f77b17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:47.278 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.276+0000 7f77bac1e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:47.278 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.276+0000 7f77bac1e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:47.278 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.276+0000 7f77bac1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:15:47.278 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.276+0000 7f77bac1e640 1 -- 172.21.15.156:0/4199054136 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77b41d7900 con 0x7f77b40b4d60 2024-03-25T18:15:47.302 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.301+0000 7f77b17fa640 1 -- 172.21.15.156:0/4199054136 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f779c0522e0 con 0x7f77b40b4d60 2024-03-25T18:15:47.302 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.301+0000 7f77b17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:47.302 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.301+0000 7f77b17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:47.302 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:15:47.302 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:47.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.303+0000 7f77bac1e640 10 librados: watch_flush enter 2024-03-25T18:15:47.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.304+0000 7f77bac1e640 10 librados: watch_flush exit 2024-03-25T18:15:47.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.304+0000 7f77bac1e640 20 client.4564.objecter shutdown clearing up homeless session... 2024-03-25T18:15:47.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.304+0000 7f77bac1e640 10 client.4564.objecter successfully canceled tick 2024-03-25T18:15:47.306 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.304+0000 7f77bac1e640 1 -- 172.21.15.156:0/4199054136 >> v1:172.21.15.194:6800/2946427713 conn(0x7f779406dce0 legacy=0x7f77940701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:47.306 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.304+0000 7f77bac1e640 10 monclient: shutdown 2024-03-25T18:15:47.306 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.304+0000 7f77bac1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:47.306 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.304+0000 7f77bac1e640 1 -- 172.21.15.156:0/4199054136 >> v1:172.21.15.194:6789/0 conn(0x7f77b40b4d60 legacy=0x7f77b4132db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:47.306 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.304+0000 7f77bac1e640 1 -- 172.21.15.156:0/4199054136 shutdown_connections 2024-03-25T18:15:47.306 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.304+0000 7f77bac1e640 1 -- 172.21.15.156:0/4199054136 >> 172.21.15.156:0/4199054136 conn(0x7f77b40617a0 msgr2=0x7f77b40b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:47.306 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.304+0000 7f77bac1e640 1 -- 172.21.15.156:0/4199054136 shutdown_connections 2024-03-25T18:15:47.306 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.304+0000 7f77bac1e640 1 -- 172.21.15.156:0/4199054136 wait complete. 2024-03-25T18:15:47.306 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:47.304+0000 7f77bac1e640 1 librados: shutdown 2024-03-25T18:15:47.317 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:54.124 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:54.239 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.238+0000 7f9dc3a25640 10 monclient: get_monmap_and_config 2024-03-25T18:15:54.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.238+0000 7f9dc3a25640 10 monclient: build_initial_monmap 2024-03-25T18:15:54.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.238+0000 7f9dc3a25640 10 monclient: monmap: 2024-03-25T18:15:54.240 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:15:54.240 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:54.240 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:15:54.239108+0000 2024-03-25T18:15:54.241 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:15:54.239108+0000 2024-03-25T18:15:54.241 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:54.241 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:54.241 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:15:54.241 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:15:54.241 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:15:54.242 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:54.242 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.238+0000 7f9dc3a25640 1 Processor -- start 2024-03-25T18:15:54.242 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.239+0000 7f9dc3a25640 1 -- start start 2024-03-25T18:15:54.242 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.239+0000 7f9dc3a25640 10 monclient: init 2024-03-25T18:15:54.243 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.239+0000 7f9dc3a25640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:54.243 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.239+0000 7f9dc3a25640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:54.243 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.239+0000 7f9dc3a25640 10 monclient(hunting): picked mon.noname-b con 0x7f9dbc0b4d60 addr v1:172.21.15.156:6790/0 2024-03-25T18:15:54.243 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.239+0000 7f9dc3a25640 10 monclient(hunting): picked mon.noname-a con 0x7f9dbc0b1730 addr v1:172.21.15.156:6789/0 2024-03-25T18:15:54.243 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.239+0000 7f9dc3a25640 10 monclient(hunting): picked mon.noname-c con 0x7f9dbc14be90 addr v1:172.21.15.194:6789/0 2024-03-25T18:15:54.243 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.239+0000 7f9dc3a25640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dbc0af940 con 0x7f9dbc0b1730 2024-03-25T18:15:54.243 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.239+0000 7f9dc3a25640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dbc0b0a70 con 0x7f9dbc14be90 2024-03-25T18:15:54.243 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.239+0000 7f9dc3a25640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dbc0b0d50 con 0x7f9dbc0b4d60 2024-03-25T18:15:54.243 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.239+0000 7f9dc3a25640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:54.244 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.239+0000 7f9dc3a25640 10 monclient(hunting): authenticate will time out at 7435.550596s 2024-03-25T18:15:54.254 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.253+0000 7f9dc17c1640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f9dbc0b4d60 0x7f9dbc13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:40174/0 (socket says 172.21.15.156:40174) 2024-03-25T18:15:54.254 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.253+0000 7f9dc17c1640 1 -- 172.21.15.156:0/1573646750 learned_addr learned my addr 172.21.15.156:0/1573646750 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:15:54.324 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.323+0000 7f9db3fff640 1 -- 172.21.15.156:0/1573646750 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2036099225 0 0) 0x7f9dbc0af940 con 0x7f9dbc0b1730 2024-03-25T18:15:54.324 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.323+0000 7f9db3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:54.324 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.323+0000 7f9db3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:54.324 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.323+0000 7f9db3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:54.324 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.323+0000 7f9db3fff640 10 monclient(hunting): my global_id is 4560 2024-03-25T18:15:54.324 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.323+0000 7f9db3fff640 1 -- 172.21.15.156:0/1573646750 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9da8003520 con 0x7f9dbc0b1730 2024-03-25T18:15:54.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.349+0000 7f9db3fff640 1 -- 172.21.15.156:0/1573646750 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3655793316 0 0) 0x7f9da8003520 con 0x7f9dbc0b1730 2024-03-25T18:15:54.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.350+0000 7f9db3fff640 1 -- 172.21.15.156:0/1573646750 >> v1:172.21.15.156:6790/0 conn(0x7f9dbc0b4d60 legacy=0x7f9dbc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:54.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.350+0000 7f9db3fff640 1 -- 172.21.15.156:0/1573646750 >> v1:172.21.15.194:6789/0 conn(0x7f9dbc14be90 legacy=0x7f9dbc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:54.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.350+0000 7f9db3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:15:54.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.350+0000 7f9db3fff640 1 monclient: found mon.noname-a 2024-03-25T18:15:54.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.350+0000 7f9db3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:54.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.350+0000 7f9db3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:15:54.351 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.350+0000 7f9db3fff640 1 -- 172.21.15.156:0/1573646750 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dbc0b1030 con 0x7f9dbc0b1730 2024-03-25T18:15:54.352 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.350+0000 7f9db3fff640 10 monclient: _finish_auth 0 2024-03-25T18:15:54.352 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.350+0000 7f9db3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:15:54.352 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.350+0000 7f9db3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:54.352 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.350+0000 7f9db3fff640 1 -- 172.21.15.156:0/1573646750 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9dac001690 con 0x7f9dbc0b1730 2024-03-25T18:15:54.352 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.350+0000 7f9dc3a25640 5 monclient: authenticate success, global_id 4560 2024-03-25T18:15:54.352 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.350+0000 7f9db3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:54.352 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.350+0000 7f9db3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:54.352 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.350+0000 7f9db3fff640 10 monclient: dump: 2024-03-25T18:15:54.352 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:54.352 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:54.352 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:54.352 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:54.352 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:54.352 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:54.352 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:54.352 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:54.353 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:54.353 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:54.353 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.350+0000 7f9dc3a25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:54.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.380+0000 7f9db3fff640 1 -- 172.21.15.156:0/1573646750 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9dac0017f0 con 0x7f9dbc0b1730 2024-03-25T18:15:54.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.380+0000 7f9db3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:54.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.380+0000 7f9db3fff640 1 -- 172.21.15.156:0/1573646750 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9dac001c00 con 0x7f9dbc0b1730 2024-03-25T18:15:54.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.380+0000 7f9db3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:54.382 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.380+0000 7f9db3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:54.382 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.380+0000 7f9db3fff640 10 monclient: dump: 2024-03-25T18:15:54.382 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:54.382 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:54.382 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:54.382 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:54.382 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:54.382 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:54.382 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:54.382 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:54.382 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:54.382 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:54.382 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.380+0000 7f9dc3a25640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:54.383 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.380+0000 7f9dc3a25640 4 set_mon_vals no callback set 2024-03-25T18:15:54.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.381+0000 7f9dc3a25640 10 monclient: shutdown 2024-03-25T18:15:54.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.381+0000 7f9dc3a25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:54.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.381+0000 7f9dc3a25640 1 -- 172.21.15.156:0/1573646750 >> v1:172.21.15.156:6789/0 conn(0x7f9dbc0b1730 legacy=0x7f9dbc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:54.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.381+0000 7f9dc3a25640 1 -- 172.21.15.156:0/1573646750 shutdown_connections 2024-03-25T18:15:54.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.381+0000 7f9dc3a25640 1 -- 172.21.15.156:0/1573646750 >> 172.21.15.156:0/1573646750 conn(0x7f9dbc0617a0 msgr2=0x7f9dbc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:54.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.381+0000 7f9dc3a25640 1 -- 172.21.15.156:0/1573646750 shutdown_connections 2024-03-25T18:15:54.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.381+0000 7f9dc3a25640 1 -- 172.21.15.156:0/1573646750 wait complete. 2024-03-25T18:15:54.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.382+0000 7f9dc3a25640 10 monclient: build_initial_monmap 2024-03-25T18:15:54.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.382+0000 7f9dc3a25640 10 monclient: monmap: 2024-03-25T18:15:54.385 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:15:54.385 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:54.385 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:15:54.386 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:15:54.386 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:54.386 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:15:54.386 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:15:54.386 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:15:54.386 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:15:54.386 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:54.386 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.382+0000 7f9dc3a25640 1 librados: starting msgr at 2024-03-25T18:15:54.386 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.382+0000 7f9dc3a25640 1 librados: starting objecter 2024-03-25T18:15:54.386 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.382+0000 7f9dc3a25640 1 Processor -- start 2024-03-25T18:15:54.386 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.382+0000 7f9dc3a25640 1 -- start start 2024-03-25T18:15:54.386 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.382+0000 7f9dc3a25640 1 librados: setting wanted keys 2024-03-25T18:15:54.386 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.382+0000 7f9dc3a25640 1 librados: calling monclient init 2024-03-25T18:15:54.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.382+0000 7f9dc3a25640 10 monclient: init 2024-03-25T18:15:54.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.383+0000 7f9dc3a25640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:54.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.383+0000 7f9dc3a25640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:15:54.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.383+0000 7f9dc3a25640 10 monclient(hunting): picked mon.noname-b con 0x7f9dbc0b1730 addr v1:172.21.15.194:6789/0 2024-03-25T18:15:54.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.383+0000 7f9dc3a25640 10 monclient(hunting): picked mon.noname-a con 0x7f9dbc0b4d60 addr v1:172.21.15.156:6789/0 2024-03-25T18:15:54.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.383+0000 7f9dc3a25640 10 monclient(hunting): picked mon.noname-c con 0x7f9dbc14be90 addr v1:172.21.15.156:6790/0 2024-03-25T18:15:54.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.383+0000 7f9dc3a25640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dbc1d7940 con 0x7f9dbc0b4d60 2024-03-25T18:15:54.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.383+0000 7f9dc3a25640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dbc1d8570 con 0x7f9dbc0b1730 2024-03-25T18:15:54.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.383+0000 7f9dc3a25640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dbc1d8700 con 0x7f9dbc14be90 2024-03-25T18:15:54.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.383+0000 7f9dc3a25640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:54.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.383+0000 7f9dc3a25640 10 monclient(hunting): authenticate will time out at 7435.693945s 2024-03-25T18:15:54.396 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.395+0000 7f9dc0fc0640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f9dbc0b4d60 0x7f9dbc136e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:50888/0 (socket says 172.21.15.156:50888) 2024-03-25T18:15:54.396 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.395+0000 7f9dc0fc0640 1 -- 172.21.15.156:0/45709133 learned_addr learned my addr 172.21.15.156:0/45709133 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:15:54.399 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.398+0000 7f9db1ffb640 10 client.?.objecter ms_handle_connect 0x7f9dbc14be90 2024-03-25T18:15:54.399 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.398+0000 7f9db1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:54.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.399+0000 7f9db1ffb640 10 client.?.objecter ms_handle_connect 0x7f9dbc0b4d60 2024-03-25T18:15:54.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.399+0000 7f9db1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:54.405 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.404+0000 7f9db1ffb640 10 client.?.objecter ms_handle_connect 0x7f9dbc0b1730 2024-03-25T18:15:54.405 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.404+0000 7f9db1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:54.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.408+0000 7f9db1ffb640 1 -- 172.21.15.156:0/45709133 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820370852 0 0) 0x7f9dbc1d8570 con 0x7f9dbc0b1730 2024-03-25T18:15:54.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.408+0000 7f9db1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:54.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.408+0000 7f9db1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:54.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.408+0000 7f9db1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:54.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.408+0000 7f9db1ffb640 10 monclient(hunting): my global_id is 4570 2024-03-25T18:15:54.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.408+0000 7f9db1ffb640 1 -- 172.21.15.156:0/45709133 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9da0004eb0 con 0x7f9dbc0b1730 2024-03-25T18:15:54.412 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.411+0000 7f9db1ffb640 1 -- 172.21.15.156:0/45709133 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927267472 0 0) 0x7f9dbc1d7940 con 0x7f9dbc0b4d60 2024-03-25T18:15:54.412 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.411+0000 7f9db1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:54.412 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.411+0000 7f9db1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:54.412 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.411+0000 7f9db1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:54.412 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.411+0000 7f9db1ffb640 10 monclient(hunting): my global_id is 4563 2024-03-25T18:15:54.413 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.411+0000 7f9db1ffb640 1 -- 172.21.15.156:0/45709133 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dbc1d8570 con 0x7f9dbc0b4d60 2024-03-25T18:15:54.417 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.416+0000 7f9db1ffb640 1 -- 172.21.15.156:0/45709133 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1829899709 0 0) 0x7f9dbc1d8700 con 0x7f9dbc14be90 2024-03-25T18:15:54.417 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.416+0000 7f9db1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:54.417 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.416+0000 7f9db1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:54.417 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.416+0000 7f9db1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:54.417 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.416+0000 7f9db1ffb640 10 monclient(hunting): my global_id is 4574 2024-03-25T18:15:54.417 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.416+0000 7f9db1ffb640 1 -- 172.21.15.156:0/45709133 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dbc1d7940 con 0x7f9dbc14be90 2024-03-25T18:15:54.421 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.420+0000 7f9db1ffb640 1 -- 172.21.15.156:0/45709133 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1594905326 0 0) 0x7f9da0004eb0 con 0x7f9dbc0b1730 2024-03-25T18:15:54.421 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.420+0000 7f9db1ffb640 1 -- 172.21.15.156:0/45709133 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9dbc1d8700 con 0x7f9dbc0b1730 2024-03-25T18:15:54.421 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.420+0000 7f9db1ffb640 1 -- 172.21.15.156:0/45709133 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9db80018a0 con 0x7f9dbc0b1730 2024-03-25T18:15:54.422 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.420+0000 7f9db1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:54.422 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.420+0000 7f9db1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:54.422 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.420+0000 7f9db1ffb640 10 monclient(hunting): dump: 2024-03-25T18:15:54.422 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:54.422 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:54.422 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:54.422 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:54.422 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:54.422 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:54.422 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:54.422 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:54.422 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:54.422 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:54.423 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.420+0000 7f9db1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:54.424 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.422+0000 7f9db1ffb640 1 -- 172.21.15.156:0/45709133 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 28535723 0 0) 0x7f9dbc1d8570 con 0x7f9dbc0b4d60 2024-03-25T18:15:54.424 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.422+0000 7f9db1ffb640 1 -- 172.21.15.156:0/45709133 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9da0004eb0 con 0x7f9dbc0b4d60 2024-03-25T18:15:54.424 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.422+0000 7f9db1ffb640 1 -- 172.21.15.156:0/45709133 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9dac001bc0 con 0x7f9dbc0b4d60 2024-03-25T18:15:54.424 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.422+0000 7f9db1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:54.424 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.422+0000 7f9db1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:54.424 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.422+0000 7f9db1ffb640 10 monclient(hunting): dump: 2024-03-25T18:15:54.424 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:54.424 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:54.424 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:54.425 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:54.425 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:54.425 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:54.425 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:54.425 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:54.425 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:54.425 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:54.431 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.430+0000 7f9db1ffb640 1 -- 172.21.15.156:0/45709133 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2469084980 0 0) 0x7f9dbc1d8700 con 0x7f9dbc0b1730 2024-03-25T18:15:54.431 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.430+0000 7f9db1ffb640 1 -- 172.21.15.156:0/45709133 >> v1:172.21.15.156:6790/0 conn(0x7f9dbc14be90 legacy=0x7f9dbc12fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:54.431 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.430+0000 7f9db1ffb640 1 -- 172.21.15.156:0/45709133 >> v1:172.21.15.156:6789/0 conn(0x7f9dbc0b4d60 legacy=0x7f9dbc136e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:54.431 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.430+0000 7f9db1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:15:54.431 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.430+0000 7f9db1ffb640 1 monclient: found mon.b 2024-03-25T18:15:54.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.430+0000 7f9db1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:54.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.430+0000 7f9db1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:15:54.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.430+0000 7f9db1ffb640 1 -- 172.21.15.156:0/45709133 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dbc1361c0 con 0x7f9dbc0b1730 2024-03-25T18:15:54.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.430+0000 7f9db1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:15:54.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.430+0000 7f9db1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:15:54.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.430+0000 7f9db1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:54.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.430+0000 7f9dc3a25640 5 monclient: authenticate success, global_id 4570 2024-03-25T18:15:54.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.430+0000 7f9dc3a25640 10 monclient: _renew_subs 2024-03-25T18:15:54.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.430+0000 7f9dc3a25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:15:54.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.430+0000 7f9dc3a25640 1 -- 172.21.15.156:0/45709133 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dbc136470 con 0x7f9dbc0b1730 2024-03-25T18:15:54.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.430+0000 7f9dc3a25640 10 client.4570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:54.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.430+0000 7f9dc3a25640 10 monclient: _renew_subs 2024-03-25T18:15:54.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.430+0000 7f9dc3a25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:15:54.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.430+0000 7f9dc3a25640 1 -- 172.21.15.156:0/45709133 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dbc137780 con 0x7f9dbc0b1730 2024-03-25T18:15:54.433 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.430+0000 7f9dc3a25640 1 librados: init done 2024-03-25T18:15:54.433 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.431+0000 7f9dc3a25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:54.433 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.431+0000 7f9dc3a25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:54.433 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.431+0000 7f9dc3a25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:15:54.433 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.431+0000 7f9dc3a25640 1 -- 172.21.15.156:0/45709133 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dbc1d77a0 con 0x7f9dbc0b1730 2024-03-25T18:15:54.725 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.724+0000 7f9db1ffb640 1 -- 172.21.15.156:0/45709133 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9db8001d10 con 0x7f9dbc0b1730 2024-03-25T18:15:54.726 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.724+0000 7f9db1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:54.726 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.724+0000 7f9db1ffb640 1 -- 172.21.15.156:0/45709133 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9db8004860 con 0x7f9dbc0b1730 2024-03-25T18:15:54.726 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.724+0000 7f9db1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:54.726 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.724+0000 7f9db1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:54.726 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.724+0000 7f9db1ffb640 10 monclient: dump: 2024-03-25T18:15:54.726 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:15:54.726 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:15:54.726 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:54.726 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:15:54.726 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:54.726 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:15:54.726 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:15:54.727 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:15:54.727 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:15:54.727 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:15:54.727 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.724+0000 7f9db37fe640 4 set_mon_vals no callback set 2024-03-25T18:15:54.727 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.725+0000 7f9db1ffb640 1 -- 172.21.15.156:0/45709133 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f9db801a8b0 con 0x7f9dbc0b1730 2024-03-25T18:15:54.727 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.725+0000 7f9dc0fc0640 10 monclient: get_auth_request con 0x7f9da006df20 auth_method 0 2024-03-25T18:15:54.727 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.726+0000 7f9db1ffb640 10 client.4570.objecter ms_handle_connect 0x7f9da006df20 2024-03-25T18:15:54.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.729+0000 7f9db1ffb640 1 -- 172.21.15.156:0/45709133 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f9db80545c0 con 0x7f9dbc0b1730 2024-03-25T18:15:54.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.729+0000 7f9db1ffb640 10 client.4570.objecter ms_dispatch 0x7f9dbc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:54.731 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.729+0000 7f9db1ffb640 3 client.4570.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:54.731 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.729+0000 7f9db1ffb640 3 client.4570.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:54.731 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.729+0000 7f9db1ffb640 20 client.4570.objecter dump_active .. 0 homeless 2024-03-25T18:15:54.731 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.730+0000 7f9db1ffb640 1 -- 172.21.15.156:0/45709133 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9db804ff00 con 0x7f9dbc0b1730 2024-03-25T18:15:54.731 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.730+0000 7f9db1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:54.731 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.730+0000 7f9db1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:54.989 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.988+0000 7f9dc3a25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:54.989 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.988+0000 7f9dc3a25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:54.989 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.988+0000 7f9dc3a25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:15:54.990 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:54.988+0000 7f9dc3a25640 1 -- 172.21.15.156:0/45709133 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9dbc1d7ad0 con 0x7f9dbc0b1730 2024-03-25T18:15:55.025 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:55.024+0000 7f9db1ffb640 1 -- 172.21.15.156:0/45709133 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f9db8052000 con 0x7f9dbc0b1730 2024-03-25T18:15:55.025 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:55.024+0000 7f9db1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:55.025 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:55.024+0000 7f9db1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:55.025 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:15:55.025 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:55.028 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:55.026+0000 7f9dc3a25640 10 librados: watch_flush enter 2024-03-25T18:15:55.028 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:55.026+0000 7f9dc3a25640 10 librados: watch_flush exit 2024-03-25T18:15:55.028 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:55.026+0000 7f9dc3a25640 20 client.4570.objecter shutdown clearing up homeless session... 2024-03-25T18:15:55.028 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:55.026+0000 7f9dc3a25640 10 client.4570.objecter successfully canceled tick 2024-03-25T18:15:55.028 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:55.027+0000 7f9dc3a25640 1 -- 172.21.15.156:0/45709133 >> v1:172.21.15.194:6800/2946427713 conn(0x7f9da006df20 legacy=0x7f9da00703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:55.028 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:55.027+0000 7f9dc3a25640 10 monclient: shutdown 2024-03-25T18:15:55.028 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:55.027+0000 7f9dc3a25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:55.028 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:55.027+0000 7f9dc3a25640 1 -- 172.21.15.156:0/45709133 >> v1:172.21.15.194:6789/0 conn(0x7f9dbc0b1730 legacy=0x7f9dbc1d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:55.028 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:55.027+0000 7f9dc3a25640 1 -- 172.21.15.156:0/45709133 shutdown_connections 2024-03-25T18:15:55.029 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:55.027+0000 7f9dc3a25640 1 -- 172.21.15.156:0/45709133 >> 172.21.15.156:0/45709133 conn(0x7f9dbc0617a0 msgr2=0x7f9dbc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:55.029 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:55.027+0000 7f9dc3a25640 1 -- 172.21.15.156:0/45709133 shutdown_connections 2024-03-25T18:15:55.029 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:55.027+0000 7f9dc3a25640 1 -- 172.21.15.156:0/45709133 wait complete. 2024-03-25T18:15:55.029 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:15:55.027+0000 7f9dc3a25640 1 librados: shutdown 2024-03-25T18:15:55.040 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:59.634 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:08:05.622 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:59.641 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:08:05.631 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:59.661 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:08:05.652 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:59.696 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:08:05.663 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:59.708 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:08:05.675 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:59.719 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:08:05.709 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:59.730 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:08:05.720 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:59.750 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:08:05.717 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:59.759 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:08:05.749 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:01.847 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:01.964 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.963+0000 7fa76b960640 10 monclient: get_monmap_and_config 2024-03-25T18:16:01.965 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.963+0000 7fa76b960640 10 monclient: build_initial_monmap 2024-03-25T18:16:01.965 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.963+0000 7fa76b960640 10 monclient: monmap: 2024-03-25T18:16:01.965 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:16:01.965 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:01.965 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:16:01.964519+0000 2024-03-25T18:16:01.965 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:16:01.964519+0000 2024-03-25T18:16:01.965 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:01.965 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:01.965 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:16:01.965 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:16:01.965 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:16:01.965 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:01.967 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.964+0000 7fa76b960640 1 Processor -- start 2024-03-25T18:16:01.967 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.964+0000 7fa76b960640 1 -- start start 2024-03-25T18:16:01.967 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.964+0000 7fa76b960640 10 monclient: init 2024-03-25T18:16:01.967 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.965+0000 7fa76b960640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:01.967 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.965+0000 7fa76b960640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:01.968 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.965+0000 7fa76b960640 10 monclient(hunting): picked mon.noname-a con 0x7fa7640a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:16:01.968 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.965+0000 7fa76b960640 10 monclient(hunting): picked mon.noname-b con 0x7fa76413f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:16:01.968 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.965+0000 7fa76b960640 10 monclient(hunting): picked mon.noname-c con 0x7fa7641430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:16:01.968 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.965+0000 7fa76b960640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa764142520 con 0x7fa7640a98f0 2024-03-25T18:16:01.968 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.965+0000 7fa76b960640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7640b28a0 con 0x7fa7641430c0 2024-03-25T18:16:01.968 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.965+0000 7fa76b960640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7640b2a30 con 0x7fa76413f710 2024-03-25T18:16:01.968 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.965+0000 7fa76b960640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:01.968 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.965+0000 7fa76b960640 10 monclient(hunting): authenticate will time out at 7443.276397s 2024-03-25T18:16:01.972 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.971+0000 7fa7696fc640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fa7640a98f0 0x7fa7640a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:36248/0 (socket says 172.21.15.156:36248) 2024-03-25T18:16:01.972 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.971+0000 7fa7696fc640 1 -- 172.21.15.156:0/4119873615 learned_addr learned my addr 172.21.15.156:0/4119873615 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:16:01.990 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.989+0000 7fa75bfff640 1 -- 172.21.15.156:0/4119873615 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3165351511 0 0) 0x7fa7640b2a30 con 0x7fa76413f710 2024-03-25T18:16:01.990 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.989+0000 7fa75bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:01.991 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.989+0000 7fa75bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:01.991 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.989+0000 7fa75bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:01.991 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.989+0000 7fa75bfff640 10 monclient(hunting): my global_id is 4577 2024-03-25T18:16:01.991 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.989+0000 7fa75bfff640 1 -- 172.21.15.156:0/4119873615 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa750003520 con 0x7fa76413f710 2024-03-25T18:16:01.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.993+0000 7fa75bfff640 1 -- 172.21.15.156:0/4119873615 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389984321 0 0) 0x7fa7640b28a0 con 0x7fa7641430c0 2024-03-25T18:16:01.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.993+0000 7fa75bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:01.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.993+0000 7fa75bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:01.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.993+0000 7fa75bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:01.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.993+0000 7fa75bfff640 10 monclient(hunting): my global_id is 4573 2024-03-25T18:16:01.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.993+0000 7fa75bfff640 1 -- 172.21.15.156:0/4119873615 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7640b2a30 con 0x7fa7641430c0 2024-03-25T18:16:01.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.993+0000 7fa75bfff640 1 -- 172.21.15.156:0/4119873615 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 453878277 0 0) 0x7fa764142520 con 0x7fa7640a98f0 2024-03-25T18:16:01.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.993+0000 7fa75bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:01.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.993+0000 7fa75bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:01.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.993+0000 7fa75bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:01.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.993+0000 7fa75bfff640 10 monclient(hunting): my global_id is 4566 2024-03-25T18:16:01.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.993+0000 7fa75bfff640 1 -- 172.21.15.156:0/4119873615 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7640b28a0 con 0x7fa7640a98f0 2024-03-25T18:16:01.999 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.998+0000 7fa75bfff640 1 -- 172.21.15.156:0/4119873615 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 680043640 0 0) 0x7fa750003520 con 0x7fa76413f710 2024-03-25T18:16:01.999 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.998+0000 7fa75bfff640 1 -- 172.21.15.156:0/4119873615 >> v1:172.21.15.194:6789/0 conn(0x7fa7641430c0 legacy=0x7fa76414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:01.999 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.998+0000 7fa75bfff640 1 -- 172.21.15.156:0/4119873615 >> v1:172.21.15.156:6789/0 conn(0x7fa7640a98f0 legacy=0x7fa7640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:02.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.998+0000 7fa75bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:02.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.998+0000 7fa75bfff640 1 monclient: found mon.noname-b 2024-03-25T18:16:02.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.998+0000 7fa75bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:02.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.998+0000 7fa75bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.156:6790/0 2024-03-25T18:16:02.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.998+0000 7fa75bfff640 1 -- 172.21.15.156:0/4119873615 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7640b44c0 con 0x7fa76413f710 2024-03-25T18:16:02.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.998+0000 7fa75bfff640 10 monclient: _finish_auth 0 2024-03-25T18:16:02.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.998+0000 7fa75bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:02.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.998+0000 7fa75bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:02.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.998+0000 7fa75bfff640 1 -- 172.21.15.156:0/4119873615 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fa754001690 con 0x7fa76413f710 2024-03-25T18:16:02.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.998+0000 7fa76b960640 5 monclient: authenticate success, global_id 4577 2024-03-25T18:16:02.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.998+0000 7fa76b960640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:02.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.998+0000 7fa75bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:02.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.998+0000 7fa75bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:02.001 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.998+0000 7fa75bfff640 10 monclient: dump: 2024-03-25T18:16:02.001 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:02.001 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:02.001 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:02.001 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:02.001 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:02.001 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:02.001 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:02.001 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:02.001 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:02.001 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:02.001 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:01.998+0000 7fa76b960640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:02.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.003+0000 7fa75bfff640 1 -- 172.21.15.156:0/4119873615 <== mon.2 v1:172.21.15.156:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7540017f0 con 0x7fa76413f710 2024-03-25T18:16:02.005 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.003+0000 7fa75bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:02.005 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.003+0000 7fa75bfff640 1 -- 172.21.15.156:0/4119873615 <== mon.2 v1:172.21.15.156:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fa754001c00 con 0x7fa76413f710 2024-03-25T18:16:02.005 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.003+0000 7fa75bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:02.005 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.003+0000 7fa75bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:02.005 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.003+0000 7fa75bfff640 10 monclient: dump: 2024-03-25T18:16:02.005 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:02.005 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:02.005 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:02.005 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:02.005 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:02.006 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:02.006 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:02.006 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:02.006 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:02.006 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:02.006 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.003+0000 7fa76b960640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:02.006 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.003+0000 7fa76b960640 4 set_mon_vals no callback set 2024-03-25T18:16:02.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.004+0000 7fa76b960640 10 monclient: shutdown 2024-03-25T18:16:02.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.004+0000 7fa76b960640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:02.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.004+0000 7fa76b960640 1 -- 172.21.15.156:0/4119873615 >> v1:172.21.15.156:6790/0 conn(0x7fa76413f710 legacy=0x7fa7641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:02.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.004+0000 7fa76b960640 1 -- 172.21.15.156:0/4119873615 shutdown_connections 2024-03-25T18:16:02.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.004+0000 7fa76b960640 1 -- 172.21.15.156:0/4119873615 >> 172.21.15.156:0/4119873615 conn(0x7fa7640b7ea0 msgr2=0x7fa7640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:02.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.004+0000 7fa76b960640 1 -- 172.21.15.156:0/4119873615 shutdown_connections 2024-03-25T18:16:02.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.004+0000 7fa76b960640 1 -- 172.21.15.156:0/4119873615 wait complete. 2024-03-25T18:16:02.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.005+0000 7fa76b960640 10 monclient: build_initial_monmap 2024-03-25T18:16:02.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.005+0000 7fa76b960640 10 monclient: monmap: 2024-03-25T18:16:02.009 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:16:02.010 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:02.010 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:16:02.010 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:16:02.010 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:02.010 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:16:02.010 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:16:02.010 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:16:02.010 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:16:02.010 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:02.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.005+0000 7fa76b960640 1 librados: starting msgr at 2024-03-25T18:16:02.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.005+0000 7fa76b960640 1 librados: starting objecter 2024-03-25T18:16:02.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.006+0000 7fa76b960640 1 Processor -- start 2024-03-25T18:16:02.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.006+0000 7fa76b960640 1 -- start start 2024-03-25T18:16:02.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.006+0000 7fa76b960640 1 librados: setting wanted keys 2024-03-25T18:16:02.011 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.006+0000 7fa76b960640 1 librados: calling monclient init 2024-03-25T18:16:02.011 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.006+0000 7fa76b960640 10 monclient: init 2024-03-25T18:16:02.011 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.006+0000 7fa76b960640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:02.011 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.006+0000 7fa76b960640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:02.011 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.006+0000 7fa76b960640 10 monclient(hunting): picked mon.noname-b con 0x7fa7640a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:16:02.011 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.006+0000 7fa76b960640 10 monclient(hunting): picked mon.noname-c con 0x7fa76413f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:16:02.011 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.006+0000 7fa76b960640 10 monclient(hunting): picked mon.noname-a con 0x7fa7641430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:16:02.011 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.006+0000 7fa76b960640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7641db9a0 con 0x7fa7641430c0 2024-03-25T18:16:02.011 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.006+0000 7fa76b960640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7641dc5d0 con 0x7fa7640a98f0 2024-03-25T18:16:02.011 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.006+0000 7fa76b960640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7641dc760 con 0x7fa76413f710 2024-03-25T18:16:02.011 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.006+0000 7fa76b960640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:02.011 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.006+0000 7fa76b960640 10 monclient(hunting): authenticate will time out at 7443.317765s 2024-03-25T18:16:02.020 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.019+0000 7fa7696fc640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fa7640a98f0 0x7fa7641dbec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:37942/0 (socket says 172.21.15.156:37942) 2024-03-25T18:16:02.020 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.019+0000 7fa7696fc640 1 -- 172.21.15.156:0/424401638 learned_addr learned my addr 172.21.15.156:0/424401638 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:16:02.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.022+0000 7fa759ffb640 10 client.?.objecter ms_handle_connect 0x7fa7640a98f0 2024-03-25T18:16:02.024 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.022+0000 7fa759ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:02.024 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.023+0000 7fa759ffb640 10 client.?.objecter ms_handle_connect 0x7fa76413f710 2024-03-25T18:16:02.024 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.023+0000 7fa759ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:02.033 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.032+0000 7fa759ffb640 10 client.?.objecter ms_handle_connect 0x7fa7641430c0 2024-03-25T18:16:02.033 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.032+0000 7fa759ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:02.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.036+0000 7fa759ffb640 1 -- 172.21.15.156:0/424401638 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 483572839 0 0) 0x7fa7641dc5d0 con 0x7fa7640a98f0 2024-03-25T18:16:02.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.036+0000 7fa759ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:02.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.036+0000 7fa759ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:02.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.036+0000 7fa759ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:02.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.036+0000 7fa759ffb640 10 monclient(hunting): my global_id is 4576 2024-03-25T18:16:02.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.036+0000 7fa759ffb640 1 -- 172.21.15.156:0/424401638 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa748004eb0 con 0x7fa7640a98f0 2024-03-25T18:16:02.041 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.040+0000 7fa759ffb640 1 -- 172.21.15.156:0/424401638 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2169530057 0 0) 0x7fa7641dc760 con 0x7fa76413f710 2024-03-25T18:16:02.041 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.040+0000 7fa759ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:02.041 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.040+0000 7fa759ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:02.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.040+0000 7fa759ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:02.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.040+0000 7fa759ffb640 10 monclient(hunting): my global_id is 4580 2024-03-25T18:16:02.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.040+0000 7fa759ffb640 1 -- 172.21.15.156:0/424401638 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7641dc5d0 con 0x7fa76413f710 2024-03-25T18:16:02.049 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.048+0000 7fa759ffb640 1 -- 172.21.15.156:0/424401638 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297259724 0 0) 0x7fa7641db9a0 con 0x7fa7641430c0 2024-03-25T18:16:02.049 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.048+0000 7fa759ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:02.049 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.048+0000 7fa759ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:02.049 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.048+0000 7fa759ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:02.049 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.048+0000 7fa759ffb640 10 monclient(hunting): my global_id is 4569 2024-03-25T18:16:02.049 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.048+0000 7fa759ffb640 1 -- 172.21.15.156:0/424401638 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7641dc760 con 0x7fa7641430c0 2024-03-25T18:16:02.057 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.056+0000 7fa759ffb640 1 -- 172.21.15.156:0/424401638 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4084638433 0 0) 0x7fa748004eb0 con 0x7fa7640a98f0 2024-03-25T18:16:02.057 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.056+0000 7fa759ffb640 1 -- 172.21.15.156:0/424401638 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7641db9a0 con 0x7fa7640a98f0 2024-03-25T18:16:02.057 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.056+0000 7fa759ffb640 1 -- 172.21.15.156:0/424401638 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fa760001910 con 0x7fa7640a98f0 2024-03-25T18:16:02.057 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.056+0000 7fa759ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:02.057 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.056+0000 7fa759ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:02.057 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.056+0000 7fa759ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:02.058 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:02.058 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:02.058 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:02.058 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:02.058 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:02.058 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:02.058 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:02.058 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:02.058 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:02.058 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:02.058 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.056+0000 7fa759ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:02.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.070+0000 7fa759ffb640 1 -- 172.21.15.156:0/424401638 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3146070462 0 0) 0x7fa7641db9a0 con 0x7fa7640a98f0 2024-03-25T18:16:02.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.070+0000 7fa759ffb640 1 -- 172.21.15.156:0/424401638 >> v1:172.21.15.156:6790/0 conn(0x7fa76413f710 legacy=0x7fa76406ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:02.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.070+0000 7fa759ffb640 1 -- 172.21.15.156:0/424401638 >> v1:172.21.15.156:6789/0 conn(0x7fa7641430c0 legacy=0x7fa7641e6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:02.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.070+0000 7fa759ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:02.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.070+0000 7fa759ffb640 1 monclient: found mon.b 2024-03-25T18:16:02.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.070+0000 7fa759ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:02.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.070+0000 7fa759ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:16:02.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.070+0000 7fa759ffb640 1 -- 172.21.15.156:0/424401638 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76406dea0 con 0x7fa7640a98f0 2024-03-25T18:16:02.072 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.070+0000 7fa759ffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:02.073 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.070+0000 7fa759ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:02.073 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.070+0000 7fa759ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:02.073 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.070+0000 7fa76b960640 5 monclient: authenticate success, global_id 4576 2024-03-25T18:16:02.073 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.070+0000 7fa76b960640 10 monclient: _renew_subs 2024-03-25T18:16:02.073 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.070+0000 7fa76b960640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:16:02.073 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.070+0000 7fa76b960640 1 -- 172.21.15.156:0/424401638 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa76406f1a0 con 0x7fa7640a98f0 2024-03-25T18:16:02.073 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.071+0000 7fa76b960640 10 client.4576.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:02.073 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.071+0000 7fa76b960640 10 monclient: _renew_subs 2024-03-25T18:16:02.073 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.071+0000 7fa76b960640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:16:02.073 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.071+0000 7fa76b960640 1 -- 172.21.15.156:0/424401638 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa764070b30 con 0x7fa7640a98f0 2024-03-25T18:16:02.073 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.071+0000 7fa76b960640 1 librados: init done 2024-03-25T18:16:02.074 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.072+0000 7fa76b960640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:02.074 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.072+0000 7fa76b960640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:02.074 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.072+0000 7fa76b960640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:16:02.074 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.072+0000 7fa76b960640 1 -- 172.21.15.156:0/424401638 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa764070660 con 0x7fa7640a98f0 2024-03-25T18:16:02.353 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.352+0000 7fa759ffb640 1 -- 172.21.15.156:0/424401638 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7600047e0 con 0x7fa7640a98f0 2024-03-25T18:16:02.354 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.352+0000 7fa759ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:02.354 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.352+0000 7fa759ffb640 1 -- 172.21.15.156:0/424401638 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fa760004bf0 con 0x7fa7640a98f0 2024-03-25T18:16:02.354 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.352+0000 7fa759ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:02.354 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.352+0000 7fa759ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:02.354 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.352+0000 7fa759ffb640 10 monclient: dump: 2024-03-25T18:16:02.354 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:02.354 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:02.354 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:02.354 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:02.354 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:02.355 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:02.355 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:02.355 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:02.355 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:02.355 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:02.355 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.352+0000 7fa75b7fe640 4 set_mon_vals no callback set 2024-03-25T18:16:02.355 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.353+0000 7fa759ffb640 1 -- 172.21.15.156:0/424401638 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7fa760004e10 con 0x7fa7640a98f0 2024-03-25T18:16:02.356 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.354+0000 7fa768efb640 10 monclient: get_auth_request con 0x7fa74806dde0 auth_method 0 2024-03-25T18:16:02.356 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.354+0000 7fa759ffb640 10 client.4576.objecter ms_handle_connect 0x7fa74806dde0 2024-03-25T18:16:02.360 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.359+0000 7fa759ffb640 1 -- 172.21.15.156:0/424401638 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7fa7600548f0 con 0x7fa7640a98f0 2024-03-25T18:16:02.360 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.359+0000 7fa759ffb640 10 client.4576.objecter ms_dispatch 0x7fa764000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:02.360 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.359+0000 7fa759ffb640 3 client.4576.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:02.360 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.359+0000 7fa759ffb640 3 client.4576.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:02.360 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.359+0000 7fa759ffb640 20 client.4576.objecter dump_active .. 0 homeless 2024-03-25T18:16:02.363 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.362+0000 7fa759ffb640 1 -- 172.21.15.156:0/424401638 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa760050400 con 0x7fa7640a98f0 2024-03-25T18:16:02.363 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.362+0000 7fa759ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:02.363 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.362+0000 7fa759ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:02.587 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.586+0000 7fa76b960640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:02.588 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.586+0000 7fa76b960640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:02.588 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.586+0000 7fa76b960640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:16:02.588 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.586+0000 7fa76b960640 1 -- 172.21.15.156:0/424401638 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7640627b0 con 0x7fa7640a98f0 2024-03-25T18:16:02.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.715+0000 7fa759ffb640 1 -- 172.21.15.156:0/424401638 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7fa760052330 con 0x7fa7640a98f0 2024-03-25T18:16:02.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.715+0000 7fa759ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:02.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.715+0000 7fa759ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:02.716 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:16:02.716 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:02.719 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.717+0000 7fa76b960640 10 librados: watch_flush enter 2024-03-25T18:16:02.719 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.717+0000 7fa76b960640 10 librados: watch_flush exit 2024-03-25T18:16:02.719 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.717+0000 7fa76b960640 20 client.4576.objecter shutdown clearing up homeless session... 2024-03-25T18:16:02.719 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.717+0000 7fa76b960640 10 client.4576.objecter successfully canceled tick 2024-03-25T18:16:02.719 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.717+0000 7fa76b960640 1 -- 172.21.15.156:0/424401638 >> v1:172.21.15.194:6800/2946427713 conn(0x7fa74806dde0 legacy=0x7fa7480702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:02.719 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.717+0000 7fa76b960640 10 monclient: shutdown 2024-03-25T18:16:02.719 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.717+0000 7fa76b960640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:02.719 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.717+0000 7fa76b960640 1 -- 172.21.15.156:0/424401638 >> v1:172.21.15.194:6789/0 conn(0x7fa7640a98f0 legacy=0x7fa7641dbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:02.720 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.717+0000 7fa76b960640 1 -- 172.21.15.156:0/424401638 shutdown_connections 2024-03-25T18:16:02.720 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.717+0000 7fa76b960640 1 -- 172.21.15.156:0/424401638 >> 172.21.15.156:0/424401638 conn(0x7fa7640b7ea0 msgr2=0x7fa764074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:02.720 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.717+0000 7fa76b960640 1 -- 172.21.15.156:0/424401638 shutdown_connections 2024-03-25T18:16:02.720 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.718+0000 7fa76b960640 1 -- 172.21.15.156:0/424401638 wait complete. 2024-03-25T18:16:02.720 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:02.718+0000 7fa76b960640 1 librados: shutdown 2024-03-25T18:16:02.731 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:09.537 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:09.651 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.649+0000 7f4ab8481640 10 monclient: get_monmap_and_config 2024-03-25T18:16:09.651 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.649+0000 7f4ab8481640 10 monclient: build_initial_monmap 2024-03-25T18:16:09.652 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.649+0000 7f4ab8481640 10 monclient: monmap: 2024-03-25T18:16:09.652 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:16:09.652 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:09.652 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:16:09.651159+0000 2024-03-25T18:16:09.652 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:16:09.651159+0000 2024-03-25T18:16:09.652 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:09.652 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:09.652 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:16:09.652 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:16:09.652 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:16:09.652 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:09.652 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.649+0000 7f4ab8481640 1 Processor -- start 2024-03-25T18:16:09.652 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.649+0000 7f4ab8481640 1 -- start start 2024-03-25T18:16:09.652 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.649+0000 7f4ab8481640 10 monclient: init 2024-03-25T18:16:09.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.650+0000 7f4ab8481640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:09.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.650+0000 7f4ab8481640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:09.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.650+0000 7f4ab8481640 10 monclient(hunting): picked mon.noname-b con 0x7f4ab00a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:16:09.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.650+0000 7f4ab8481640 10 monclient(hunting): picked mon.noname-a con 0x7f4ab013f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:16:09.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.650+0000 7f4ab8481640 10 monclient(hunting): picked mon.noname-c con 0x7f4ab01430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:16:09.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.650+0000 7f4ab8481640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab0142520 con 0x7f4ab013f710 2024-03-25T18:16:09.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.650+0000 7f4ab8481640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab00b28a0 con 0x7f4ab01430c0 2024-03-25T18:16:09.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.650+0000 7f4ab8481640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab00b2a30 con 0x7f4ab00a98f0 2024-03-25T18:16:09.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.650+0000 7f4ab8481640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:09.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.650+0000 7f4ab8481640 10 monclient(hunting): authenticate will time out at 7450.962681s 2024-03-25T18:16:09.662 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.660+0000 7f4ab5a1c640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f4ab013f710 0x7f4ab01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:41108/0 (socket says 172.21.15.156:41108) 2024-03-25T18:16:09.662 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.660+0000 7f4ab5a1c640 1 -- 172.21.15.156:0/2188888164 learned_addr learned my addr 172.21.15.156:0/2188888164 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:16:09.675 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.673+0000 7f4ab521b640 1 -- 172.21.15.156:0/2188888164 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1214170425 0 0) 0x7f4ab00b28a0 con 0x7f4ab01430c0 2024-03-25T18:16:09.675 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.673+0000 7f4ab521b640 10 monclient(hunting): _negotiate 2024-03-25T18:16:09.675 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.673+0000 7f4ab521b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:09.675 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.673+0000 7f4ab521b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:09.675 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.673+0000 7f4ab521b640 10 monclient(hunting): my global_id is 4579 2024-03-25T18:16:09.675 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.673+0000 7f4ab521b640 1 -- 172.21.15.156:0/2188888164 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a9c003520 con 0x7f4ab01430c0 2024-03-25T18:16:09.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.675+0000 7f4ab521b640 1 -- 172.21.15.156:0/2188888164 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 421885612 0 0) 0x7f4ab0142520 con 0x7f4ab013f710 2024-03-25T18:16:09.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.675+0000 7f4ab521b640 10 monclient(hunting): _negotiate 2024-03-25T18:16:09.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.675+0000 7f4ab521b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:09.678 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.675+0000 7f4ab521b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:09.678 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.675+0000 7f4ab521b640 10 monclient(hunting): my global_id is 4572 2024-03-25T18:16:09.678 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.675+0000 7f4ab521b640 1 -- 172.21.15.156:0/2188888164 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ab00b28a0 con 0x7f4ab013f710 2024-03-25T18:16:09.683 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.680+0000 7f4ab521b640 1 -- 172.21.15.156:0/2188888164 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2359537962 0 0) 0x7f4ab00b2a30 con 0x7f4ab00a98f0 2024-03-25T18:16:09.683 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.681+0000 7f4ab521b640 10 monclient(hunting): _negotiate 2024-03-25T18:16:09.683 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.681+0000 7f4ab521b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:09.683 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.681+0000 7f4ab521b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:09.683 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.681+0000 7f4ab521b640 10 monclient(hunting): my global_id is 4583 2024-03-25T18:16:09.683 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.681+0000 7f4ab521b640 1 -- 172.21.15.156:0/2188888164 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ab0142520 con 0x7f4ab00a98f0 2024-03-25T18:16:09.685 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.683+0000 7f4ab521b640 1 -- 172.21.15.156:0/2188888164 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1115137361 0 0) 0x7f4a9c003520 con 0x7f4ab01430c0 2024-03-25T18:16:09.685 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.683+0000 7f4ab521b640 1 -- 172.21.15.156:0/2188888164 >> v1:172.21.15.156:6790/0 conn(0x7f4ab00a98f0 legacy=0x7f4ab00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:09.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.683+0000 7f4ab521b640 1 -- 172.21.15.156:0/2188888164 >> v1:172.21.15.156:6789/0 conn(0x7f4ab013f710 legacy=0x7f4ab01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:09.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.683+0000 7f4ab521b640 10 monclient: _finish_hunting 0 2024-03-25T18:16:09.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.683+0000 7f4ab521b640 1 monclient: found mon.noname-c 2024-03-25T18:16:09.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.683+0000 7f4ab521b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:09.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.683+0000 7f4ab521b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:16:09.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.683+0000 7f4ab521b640 1 -- 172.21.15.156:0/2188888164 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab00b44c0 con 0x7f4ab01430c0 2024-03-25T18:16:09.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.683+0000 7f4ab521b640 10 monclient: _finish_auth 0 2024-03-25T18:16:09.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.683+0000 7f4ab521b640 10 monclient: _check_auth_tickets 2024-03-25T18:16:09.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.683+0000 7f4ab521b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:09.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.683+0000 7f4ab521b640 1 -- 172.21.15.156:0/2188888164 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f4aa8001a10 con 0x7f4ab01430c0 2024-03-25T18:16:09.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.683+0000 7f4ab521b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:09.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.683+0000 7f4ab521b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:09.686 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.683+0000 7f4ab521b640 10 monclient: dump: 2024-03-25T18:16:09.687 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:09.687 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:09.687 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:09.687 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:09.687 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:09.687 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:09.687 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:09.687 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:09.687 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:09.687 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:09.687 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.683+0000 7f4ab8481640 5 monclient: authenticate success, global_id 4579 2024-03-25T18:16:09.687 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.683+0000 7f4ab8481640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:09.696 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.694+0000 7f4ab521b640 1 -- 172.21.15.156:0/2188888164 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4aa8001b70 con 0x7f4ab01430c0 2024-03-25T18:16:09.696 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.694+0000 7f4ab521b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:09.696 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.694+0000 7f4ab521b640 1 -- 172.21.15.156:0/2188888164 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f4aa8001f80 con 0x7f4ab01430c0 2024-03-25T18:16:09.696 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.694+0000 7f4ab8481640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:09.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.694+0000 7f4ab8481640 4 set_mon_vals no callback set 2024-03-25T18:16:09.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.694+0000 7f4ab521b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:09.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.694+0000 7f4ab521b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:09.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.694+0000 7f4ab521b640 10 monclient: dump: 2024-03-25T18:16:09.697 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:09.697 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:09.697 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:09.697 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:09.697 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:09.697 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:09.697 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:09.697 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:09.697 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:09.697 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:09.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.695+0000 7f4ab8481640 10 monclient: shutdown 2024-03-25T18:16:09.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.695+0000 7f4ab8481640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:09.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.695+0000 7f4ab8481640 1 -- 172.21.15.156:0/2188888164 >> v1:172.21.15.194:6789/0 conn(0x7f4ab01430c0 legacy=0x7f4ab014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:09.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.695+0000 7f4ab8481640 1 -- 172.21.15.156:0/2188888164 shutdown_connections 2024-03-25T18:16:09.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.695+0000 7f4ab8481640 1 -- 172.21.15.156:0/2188888164 >> 172.21.15.156:0/2188888164 conn(0x7f4ab00b7ea0 msgr2=0x7f4ab00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:09.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.695+0000 7f4ab8481640 1 -- 172.21.15.156:0/2188888164 shutdown_connections 2024-03-25T18:16:09.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.695+0000 7f4ab8481640 1 -- 172.21.15.156:0/2188888164 wait complete. 2024-03-25T18:16:09.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.696+0000 7f4ab8481640 10 monclient: build_initial_monmap 2024-03-25T18:16:09.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.696+0000 7f4ab8481640 10 monclient: monmap: 2024-03-25T18:16:09.699 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:16:09.699 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:09.700 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:16:09.700 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:16:09.700 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:09.700 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:16:09.700 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:16:09.700 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:16:09.700 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:16:09.700 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:09.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.696+0000 7f4ab8481640 1 librados: starting msgr at 2024-03-25T18:16:09.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.696+0000 7f4ab8481640 1 librados: starting objecter 2024-03-25T18:16:09.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.696+0000 7f4ab8481640 1 Processor -- start 2024-03-25T18:16:09.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.697+0000 7f4ab8481640 1 -- start start 2024-03-25T18:16:09.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.697+0000 7f4ab8481640 1 librados: setting wanted keys 2024-03-25T18:16:09.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.697+0000 7f4ab8481640 1 librados: calling monclient init 2024-03-25T18:16:09.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.697+0000 7f4ab8481640 10 monclient: init 2024-03-25T18:16:09.701 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.697+0000 7f4ab8481640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:09.701 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.697+0000 7f4ab8481640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:09.701 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.697+0000 7f4ab8481640 10 monclient(hunting): picked mon.noname-b con 0x7f4ab00a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:16:09.702 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.697+0000 7f4ab8481640 10 monclient(hunting): picked mon.noname-c con 0x7f4ab013f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:16:09.702 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.697+0000 7f4ab8481640 10 monclient(hunting): picked mon.noname-a con 0x7f4ab01430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:16:09.702 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.697+0000 7f4ab8481640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab0136ec0 con 0x7f4ab01430c0 2024-03-25T18:16:09.702 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.697+0000 7f4ab8481640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab0136c20 con 0x7f4ab00a98f0 2024-03-25T18:16:09.702 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.697+0000 7f4ab8481640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab0137db0 con 0x7f4ab013f710 2024-03-25T18:16:09.702 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.697+0000 7f4ab8481640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:09.702 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.697+0000 7f4ab8481640 10 monclient(hunting): authenticate will time out at 7451.009749s 2024-03-25T18:16:09.709 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.707+0000 7f4ab621d640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f4ab00a98f0 0x7f4ab01376a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:42798/0 (socket says 172.21.15.156:42798) 2024-03-25T18:16:09.709 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.707+0000 7f4ab621d640 1 -- 172.21.15.156:0/1926086865 learned_addr learned my addr 172.21.15.156:0/1926086865 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:16:09.715 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.713+0000 7f4aa6ffd640 10 client.?.objecter ms_handle_connect 0x7f4ab01430c0 2024-03-25T18:16:09.715 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.713+0000 7f4aa6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:09.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.714+0000 7f4aa6ffd640 10 client.?.objecter ms_handle_connect 0x7f4ab013f710 2024-03-25T18:16:09.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.714+0000 7f4aa6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:09.717 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.715+0000 7f4aa6ffd640 10 client.?.objecter ms_handle_connect 0x7f4ab00a98f0 2024-03-25T18:16:09.717 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.715+0000 7f4aa6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:09.724 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.722+0000 7f4aa6ffd640 1 -- 172.21.15.156:0/1926086865 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2246878552 0 0) 0x7f4ab0136ec0 con 0x7f4ab01430c0 2024-03-25T18:16:09.724 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.722+0000 7f4aa6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:09.724 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.722+0000 7f4aa6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:09.724 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.722+0000 7f4aa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:09.724 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.722+0000 7f4aa6ffd640 10 monclient(hunting): my global_id is 4575 2024-03-25T18:16:09.724 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.722+0000 7f4aa6ffd640 1 -- 172.21.15.156:0/1926086865 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a94004eb0 con 0x7f4ab01430c0 2024-03-25T18:16:09.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.728+0000 7f4aa6ffd640 1 -- 172.21.15.156:0/1926086865 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790630488 0 0) 0x7f4ab0137db0 con 0x7f4ab013f710 2024-03-25T18:16:09.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.728+0000 7f4aa6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:09.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.728+0000 7f4aa6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:09.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.728+0000 7f4aa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:09.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.728+0000 7f4aa6ffd640 10 monclient(hunting): my global_id is 4586 2024-03-25T18:16:09.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.728+0000 7f4aa6ffd640 1 -- 172.21.15.156:0/1926086865 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ab0136ec0 con 0x7f4ab013f710 2024-03-25T18:16:09.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.732+0000 7f4aa6ffd640 1 -- 172.21.15.156:0/1926086865 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4036659959 0 0) 0x7f4ab0136c20 con 0x7f4ab00a98f0 2024-03-25T18:16:09.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.732+0000 7f4aa6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:09.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.732+0000 7f4aa6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:09.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.732+0000 7f4aa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:09.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.732+0000 7f4aa6ffd640 10 monclient(hunting): my global_id is 4582 2024-03-25T18:16:09.734 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.732+0000 7f4aa6ffd640 1 -- 172.21.15.156:0/1926086865 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ab0137db0 con 0x7f4ab00a98f0 2024-03-25T18:16:09.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.742+0000 7f4aa6ffd640 1 -- 172.21.15.156:0/1926086865 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 77770267 0 0) 0x7f4a94004eb0 con 0x7f4ab01430c0 2024-03-25T18:16:09.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.742+0000 7f4aa6ffd640 1 -- 172.21.15.156:0/1926086865 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ab0136c20 con 0x7f4ab01430c0 2024-03-25T18:16:09.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.742+0000 7f4aa6ffd640 1 -- 172.21.15.156:0/1926086865 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f4aa8001e80 con 0x7f4ab01430c0 2024-03-25T18:16:09.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.742+0000 7f4aa6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:09.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.742+0000 7f4aa6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:09.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.742+0000 7f4aa6ffd640 10 monclient(hunting): dump: 2024-03-25T18:16:09.745 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:09.745 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:09.745 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:09.745 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:09.745 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:09.745 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:09.745 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:09.745 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:09.745 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:09.745 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:09.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.742+0000 7f4aa6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:09.750 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.748+0000 7f4aa6ffd640 1 -- 172.21.15.156:0/1926086865 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 515791739 0 0) 0x7f4ab0137db0 con 0x7f4ab00a98f0 2024-03-25T18:16:09.750 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.748+0000 7f4aa6ffd640 1 -- 172.21.15.156:0/1926086865 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a94004eb0 con 0x7f4ab00a98f0 2024-03-25T18:16:09.750 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.748+0000 7f4aa6ffd640 1 -- 172.21.15.156:0/1926086865 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f4aac0019b0 con 0x7f4ab00a98f0 2024-03-25T18:16:09.750 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.748+0000 7f4aa6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:09.751 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.748+0000 7f4aa6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:09.751 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.748+0000 7f4aa6ffd640 10 monclient(hunting): dump: 2024-03-25T18:16:09.751 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:09.751 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:09.751 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:09.751 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:09.751 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:09.751 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:09.751 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:09.751 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:09.751 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:09.751 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:09.755 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.753+0000 7f4aa6ffd640 1 -- 172.21.15.156:0/1926086865 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 794489235 0 0) 0x7f4ab0136ec0 con 0x7f4ab013f710 2024-03-25T18:16:09.755 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.753+0000 7f4aa6ffd640 1 -- 172.21.15.156:0/1926086865 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ab0137db0 con 0x7f4ab013f710 2024-03-25T18:16:09.755 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.753+0000 7f4aa6ffd640 1 -- 172.21.15.156:0/1926086865 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f4aa0001970 con 0x7f4ab013f710 2024-03-25T18:16:09.756 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.753+0000 7f4aa6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:09.756 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.753+0000 7f4aa6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:09.756 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.753+0000 7f4aa6ffd640 10 monclient(hunting): dump: 2024-03-25T18:16:09.756 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:09.756 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:09.756 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:09.756 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:09.756 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:09.756 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:09.756 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:09.756 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:09.756 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:09.756 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:09.762 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.759+0000 7f4aa6ffd640 1 -- 172.21.15.156:0/1926086865 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 662188632 0 0) 0x7f4ab0136c20 con 0x7f4ab01430c0 2024-03-25T18:16:09.762 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.759+0000 7f4aa6ffd640 1 -- 172.21.15.156:0/1926086865 >> v1:172.21.15.156:6790/0 conn(0x7f4ab013f710 legacy=0x7f4ab00b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:09.762 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.759+0000 7f4aa6ffd640 1 -- 172.21.15.156:0/1926086865 >> v1:172.21.15.194:6789/0 conn(0x7f4ab00a98f0 legacy=0x7f4ab01376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:09.762 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.759+0000 7f4aa6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:09.762 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.759+0000 7f4aa6ffd640 1 monclient: found mon.a 2024-03-25T18:16:09.762 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.759+0000 7f4aa6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:09.762 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.759+0000 7f4aa6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:16:09.762 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.759+0000 7f4aa6ffd640 1 -- 172.21.15.156:0/1926086865 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab0138b50 con 0x7f4ab01430c0 2024-03-25T18:16:09.762 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.759+0000 7f4aa6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:16:09.762 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.759+0000 7f4aa6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:09.762 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.759+0000 7f4aa6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:09.763 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.759+0000 7f4ab8481640 5 monclient: authenticate success, global_id 4575 2024-03-25T18:16:09.763 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.759+0000 7f4ab8481640 10 monclient: _renew_subs 2024-03-25T18:16:09.763 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.759+0000 7f4ab8481640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:16:09.763 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.759+0000 7f4ab8481640 1 -- 172.21.15.156:0/1926086865 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ab0139830 con 0x7f4ab01430c0 2024-03-25T18:16:09.763 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.759+0000 7f4ab8481640 10 client.4575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:09.763 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.759+0000 7f4ab8481640 10 monclient: _renew_subs 2024-03-25T18:16:09.763 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.759+0000 7f4ab8481640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:16:09.763 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.759+0000 7f4ab8481640 1 -- 172.21.15.156:0/1926086865 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ab01369b0 con 0x7f4ab01430c0 2024-03-25T18:16:09.763 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.759+0000 7f4ab8481640 1 librados: init done 2024-03-25T18:16:09.763 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.761+0000 7f4ab8481640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:09.764 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.761+0000 7f4ab8481640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:09.764 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.761+0000 7f4ab8481640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:16:09.764 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:09.761+0000 7f4ab8481640 1 -- 172.21.15.156:0/1926086865 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ab0139a20 con 0x7f4ab01430c0 2024-03-25T18:16:10.059 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.057+0000 7f4aa6ffd640 1 -- 172.21.15.156:0/1926086865 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4aa8002020 con 0x7f4ab01430c0 2024-03-25T18:16:10.059 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.057+0000 7f4aa6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:10.059 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.057+0000 7f4aa6ffd640 1 -- 172.21.15.156:0/1926086865 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f4aa8003c20 con 0x7f4ab01430c0 2024-03-25T18:16:10.059 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.057+0000 7f4aa6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:10.059 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.057+0000 7f4aa6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:10.059 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.057+0000 7f4aa6ffd640 10 monclient: dump: 2024-03-25T18:16:10.059 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:10.059 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:10.060 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:10.060 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:10.060 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:10.060 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:10.060 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:10.060 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:10.060 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:10.060 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:10.060 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.057+0000 7f4ab4a1a640 4 set_mon_vals no callback set 2024-03-25T18:16:10.061 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.058+0000 7f4aa6ffd640 1 -- 172.21.15.156:0/1926086865 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f4aa8019bf0 con 0x7f4ab01430c0 2024-03-25T18:16:10.061 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.058+0000 7f4ab621d640 10 monclient: get_auth_request con 0x7f4a9406e060 auth_method 0 2024-03-25T18:16:10.061 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.059+0000 7f4aa6ffd640 10 client.4575.objecter ms_handle_connect 0x7f4a9406e060 2024-03-25T18:16:10.065 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.063+0000 7f4aa6ffd640 1 -- 172.21.15.156:0/1926086865 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f4aa8053780 con 0x7f4ab01430c0 2024-03-25T18:16:10.065 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.063+0000 7f4aa6ffd640 10 client.4575.objecter ms_dispatch 0x7f4ab0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:10.065 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.063+0000 7f4aa6ffd640 3 client.4575.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:10.065 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.063+0000 7f4aa6ffd640 3 client.4575.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:10.066 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.063+0000 7f4aa6ffd640 20 client.4575.objecter dump_active .. 0 homeless 2024-03-25T18:16:10.066 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.063+0000 7f4aa6ffd640 1 -- 172.21.15.156:0/1926086865 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4aa804f290 con 0x7f4ab01430c0 2024-03-25T18:16:10.066 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.063+0000 7f4aa6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:10.066 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.063+0000 7f4aa6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:10.315 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.312+0000 7f4ab8481640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:10.315 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.312+0000 7f4ab8481640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:10.315 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.312+0000 7f4ab8481640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:16:10.315 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.312+0000 7f4ab8481640 1 -- 172.21.15.156:0/1926086865 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ab014f320 con 0x7f4ab01430c0 2024-03-25T18:16:10.357 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.355+0000 7f4aa6ffd640 1 -- 172.21.15.156:0/1926086865 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f4aa80511c0 con 0x7f4ab01430c0 2024-03-25T18:16:10.358 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.355+0000 7f4aa6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:10.358 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.355+0000 7f4aa6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:10.358 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:16:10.358 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:10.360 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.357+0000 7f4ab8481640 10 librados: watch_flush enter 2024-03-25T18:16:10.361 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.357+0000 7f4ab8481640 10 librados: watch_flush exit 2024-03-25T18:16:10.361 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.357+0000 7f4ab8481640 20 client.4575.objecter shutdown clearing up homeless session... 2024-03-25T18:16:10.361 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.357+0000 7f4ab8481640 10 client.4575.objecter successfully canceled tick 2024-03-25T18:16:10.361 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.358+0000 7f4ab8481640 1 -- 172.21.15.156:0/1926086865 >> v1:172.21.15.194:6800/2946427713 conn(0x7f4a9406e060 legacy=0x7f4a94070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:10.361 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.358+0000 7f4ab8481640 10 monclient: shutdown 2024-03-25T18:16:10.361 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.358+0000 7f4ab8481640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:10.361 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.358+0000 7f4ab8481640 1 -- 172.21.15.156:0/1926086865 >> v1:172.21.15.156:6789/0 conn(0x7f4ab01430c0 legacy=0x7f4ab01e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:10.361 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.358+0000 7f4ab8481640 1 -- 172.21.15.156:0/1926086865 shutdown_connections 2024-03-25T18:16:10.361 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.358+0000 7f4ab8481640 1 -- 172.21.15.156:0/1926086865 >> 172.21.15.156:0/1926086865 conn(0x7f4ab00b7ea0 msgr2=0x7f4ab013ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:10.361 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.358+0000 7f4ab8481640 1 -- 172.21.15.156:0/1926086865 shutdown_connections 2024-03-25T18:16:10.361 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.358+0000 7f4ab8481640 1 -- 172.21.15.156:0/1926086865 wait complete. 2024-03-25T18:16:10.361 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:10.358+0000 7f4ab8481640 1 librados: shutdown 2024-03-25T18:16:10.370 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:17.176 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:17.292 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.291+0000 7fd282674640 10 monclient: get_monmap_and_config 2024-03-25T18:16:17.292 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.291+0000 7fd282674640 10 monclient: build_initial_monmap 2024-03-25T18:16:17.292 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.291+0000 7fd282674640 10 monclient: monmap: 2024-03-25T18:16:17.292 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:16:17.292 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:17.292 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:16:17.292309+0000 2024-03-25T18:16:17.292 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:16:17.292309+0000 2024-03-25T18:16:17.292 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:17.292 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:17.293 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:16:17.293 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:16:17.293 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:16:17.293 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:17.294 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.292+0000 7fd282674640 1 Processor -- start 2024-03-25T18:16:17.294 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.292+0000 7fd282674640 1 -- start start 2024-03-25T18:16:17.294 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.292+0000 7fd282674640 10 monclient: init 2024-03-25T18:16:17.294 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.293+0000 7fd282674640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:17.294 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.293+0000 7fd282674640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:17.295 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.293+0000 7fd282674640 10 monclient(hunting): picked mon.noname-a con 0x7fd27c0a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:16:17.295 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.293+0000 7fd282674640 10 monclient(hunting): picked mon.noname-b con 0x7fd27c13f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:16:17.295 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.293+0000 7fd282674640 10 monclient(hunting): picked mon.noname-c con 0x7fd27c1430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:16:17.295 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.293+0000 7fd282674640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd27c142520 con 0x7fd27c0a98f0 2024-03-25T18:16:17.295 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.293+0000 7fd282674640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd27c0b28a0 con 0x7fd27c1430c0 2024-03-25T18:16:17.295 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.293+0000 7fd282674640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd27c0b2a30 con 0x7fd27c13f710 2024-03-25T18:16:17.295 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.293+0000 7fd282674640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:17.295 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.293+0000 7fd282674640 10 monclient(hunting): authenticate will time out at 7458.604019s 2024-03-25T18:16:17.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.304+0000 7fd27bfff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fd27c0a98f0 0x7fd27c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:40364/0 (socket says 172.21.15.156:40364) 2024-03-25T18:16:17.306 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.305+0000 7fd27bfff640 1 -- 172.21.15.156:0/264295218 learned_addr learned my addr 172.21.15.156:0/264295218 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:16:17.312 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.311+0000 7fd27affd640 1 -- 172.21.15.156:0/264295218 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 25179893 0 0) 0x7fd27c142520 con 0x7fd27c0a98f0 2024-03-25T18:16:17.312 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.311+0000 7fd27affd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:17.312 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.311+0000 7fd27affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:17.312 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.311+0000 7fd27affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:17.312 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.311+0000 7fd27affd640 10 monclient(hunting): my global_id is 4578 2024-03-25T18:16:17.312 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.311+0000 7fd27affd640 1 -- 172.21.15.156:0/264295218 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd264003520 con 0x7fd27c0a98f0 2024-03-25T18:16:17.319 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.318+0000 7fd27affd640 1 -- 172.21.15.156:0/264295218 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1237140559 0 0) 0x7fd264003520 con 0x7fd27c0a98f0 2024-03-25T18:16:17.319 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.318+0000 7fd27affd640 1 -- 172.21.15.156:0/264295218 >> v1:172.21.15.156:6790/0 conn(0x7fd27c13f710 legacy=0x7fd27c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.319 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.318+0000 7fd27affd640 1 -- 172.21.15.156:0/264295218 >> v1:172.21.15.194:6789/0 conn(0x7fd27c1430c0 legacy=0x7fd27c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.319 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.318+0000 7fd27affd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:17.319 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.318+0000 7fd27affd640 1 monclient: found mon.noname-a 2024-03-25T18:16:17.319 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.318+0000 7fd27affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:17.319 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.318+0000 7fd27affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:16:17.319 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.318+0000 7fd27affd640 1 -- 172.21.15.156:0/264295218 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd27c0b44c0 con 0x7fd27c0a98f0 2024-03-25T18:16:17.320 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.318+0000 7fd27affd640 10 monclient: _finish_auth 0 2024-03-25T18:16:17.320 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.318+0000 7fd27affd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:17.320 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.318+0000 7fd27affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:17.320 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.318+0000 7fd27affd640 1 -- 172.21.15.156:0/264295218 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fd26c001690 con 0x7fd27c0a98f0 2024-03-25T18:16:17.320 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.318+0000 7fd282674640 5 monclient: authenticate success, global_id 4578 2024-03-25T18:16:17.320 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.318+0000 7fd282674640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:17.320 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.318+0000 7fd27affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:17.320 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.318+0000 7fd27affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:17.320 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.318+0000 7fd27affd640 10 monclient: dump: 2024-03-25T18:16:17.320 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:17.320 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:17.320 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:17.320 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:17.320 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:17.320 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:17.320 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:17.321 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:17.321 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:17.321 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:17.321 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.318+0000 7fd282674640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:17.322 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.321+0000 7fd27affd640 1 -- 172.21.15.156:0/264295218 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd26c0017f0 con 0x7fd27c0a98f0 2024-03-25T18:16:17.322 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.321+0000 7fd27affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:17.322 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.321+0000 7fd27affd640 1 -- 172.21.15.156:0/264295218 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fd26c001c40 con 0x7fd27c0a98f0 2024-03-25T18:16:17.322 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.321+0000 7fd282674640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:17.322 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.321+0000 7fd282674640 4 set_mon_vals no callback set 2024-03-25T18:16:17.322 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.321+0000 7fd27affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:17.322 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.321+0000 7fd27affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:17.322 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.321+0000 7fd27affd640 10 monclient: dump: 2024-03-25T18:16:17.322 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:17.322 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:17.323 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:17.323 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:17.323 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:17.323 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:17.323 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:17.323 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:17.323 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:17.323 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:17.325 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.321+0000 7fd282674640 10 monclient: shutdown 2024-03-25T18:16:17.325 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.321+0000 7fd282674640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:17.325 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.321+0000 7fd282674640 1 -- 172.21.15.156:0/264295218 >> v1:172.21.15.156:6789/0 conn(0x7fd27c0a98f0 legacy=0x7fd27c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.325 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.321+0000 7fd282674640 1 -- 172.21.15.156:0/264295218 shutdown_connections 2024-03-25T18:16:17.325 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.321+0000 7fd282674640 1 -- 172.21.15.156:0/264295218 >> 172.21.15.156:0/264295218 conn(0x7fd27c0b7ea0 msgr2=0x7fd27c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:17.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.322+0000 7fd282674640 1 -- 172.21.15.156:0/264295218 shutdown_connections 2024-03-25T18:16:17.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.322+0000 7fd282674640 1 -- 172.21.15.156:0/264295218 wait complete. 2024-03-25T18:16:17.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.322+0000 7fd282674640 10 monclient: build_initial_monmap 2024-03-25T18:16:17.326 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.322+0000 7fd282674640 10 monclient: monmap: 2024-03-25T18:16:17.326 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:16:17.326 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:17.326 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:16:17.326 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:16:17.326 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:17.326 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:16:17.326 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:16:17.326 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:16:17.326 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:16:17.326 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:17.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.323+0000 7fd282674640 1 librados: starting msgr at 2024-03-25T18:16:17.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.323+0000 7fd282674640 1 librados: starting objecter 2024-03-25T18:16:17.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.323+0000 7fd282674640 1 Processor -- start 2024-03-25T18:16:17.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.323+0000 7fd282674640 1 -- start start 2024-03-25T18:16:17.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.323+0000 7fd282674640 1 librados: setting wanted keys 2024-03-25T18:16:17.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.323+0000 7fd282674640 1 librados: calling monclient init 2024-03-25T18:16:17.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.323+0000 7fd282674640 10 monclient: init 2024-03-25T18:16:17.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.324+0000 7fd282674640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:17.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.324+0000 7fd282674640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:17.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.324+0000 7fd282674640 10 monclient(hunting): picked mon.noname-b con 0x7fd27c0a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:16:17.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.324+0000 7fd282674640 10 monclient(hunting): picked mon.noname-c con 0x7fd27c13f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:16:17.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.324+0000 7fd282674640 10 monclient(hunting): picked mon.noname-a con 0x7fd27c1430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:16:17.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.324+0000 7fd282674640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd27c1d82d0 con 0x7fd27c1430c0 2024-03-25T18:16:17.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.324+0000 7fd282674640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd27c0abb90 con 0x7fd27c0a98f0 2024-03-25T18:16:17.328 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.324+0000 7fd282674640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd27c1d76a0 con 0x7fd27c13f710 2024-03-25T18:16:17.328 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.324+0000 7fd282674640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:17.328 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.324+0000 7fd282674640 10 monclient(hunting): authenticate will time out at 7458.635094s 2024-03-25T18:16:17.328 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.327+0000 7fd280c11640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fd27c1430c0 0x7fd27c134010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:40380/0 (socket says 172.21.15.156:40380) 2024-03-25T18:16:17.328 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.327+0000 7fd280c11640 1 -- 172.21.15.156:0/2184885762 learned_addr learned my addr 172.21.15.156:0/2184885762 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:16:17.331 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.329+0000 7fd278ff9640 10 client.?.objecter ms_handle_connect 0x7fd27c1430c0 2024-03-25T18:16:17.331 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.329+0000 7fd278ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:17.331 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.330+0000 7fd278ff9640 10 client.?.objecter ms_handle_connect 0x7fd27c0a98f0 2024-03-25T18:16:17.331 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.330+0000 7fd278ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:17.332 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.331+0000 7fd278ff9640 10 client.?.objecter ms_handle_connect 0x7fd27c13f710 2024-03-25T18:16:17.332 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.331+0000 7fd278ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:17.342 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.341+0000 7fd278ff9640 1 -- 172.21.15.156:0/2184885762 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 619707265 0 0) 0x7fd27c1d82d0 con 0x7fd27c1430c0 2024-03-25T18:16:17.342 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.341+0000 7fd278ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:17.342 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.341+0000 7fd278ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:17.342 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.341+0000 7fd278ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:17.342 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.341+0000 7fd278ff9640 10 monclient(hunting): my global_id is 4581 2024-03-25T18:16:17.342 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.341+0000 7fd278ff9640 1 -- 172.21.15.156:0/2184885762 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd268004eb0 con 0x7fd27c1430c0 2024-03-25T18:16:17.347 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.346+0000 7fd278ff9640 1 -- 172.21.15.156:0/2184885762 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2546472353 0 0) 0x7fd27c1d76a0 con 0x7fd27c13f710 2024-03-25T18:16:17.347 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.346+0000 7fd278ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:17.347 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.346+0000 7fd278ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:17.347 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.346+0000 7fd278ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:17.347 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.346+0000 7fd278ff9640 10 monclient(hunting): my global_id is 4592 2024-03-25T18:16:17.347 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.346+0000 7fd278ff9640 1 -- 172.21.15.156:0/2184885762 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd27c1d82d0 con 0x7fd27c13f710 2024-03-25T18:16:17.349 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.348+0000 7fd278ff9640 1 -- 172.21.15.156:0/2184885762 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245646259 0 0) 0x7fd27c0abb90 con 0x7fd27c0a98f0 2024-03-25T18:16:17.349 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.348+0000 7fd278ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:17.349 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.348+0000 7fd278ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:17.349 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.348+0000 7fd278ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:17.349 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.348+0000 7fd278ff9640 10 monclient(hunting): my global_id is 4588 2024-03-25T18:16:17.349 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.348+0000 7fd278ff9640 1 -- 172.21.15.156:0/2184885762 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd27c1d76a0 con 0x7fd27c0a98f0 2024-03-25T18:16:17.361 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.360+0000 7fd278ff9640 1 -- 172.21.15.156:0/2184885762 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4135866381 0 0) 0x7fd268004eb0 con 0x7fd27c1430c0 2024-03-25T18:16:17.361 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.360+0000 7fd278ff9640 1 -- 172.21.15.156:0/2184885762 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd27c0abb90 con 0x7fd27c1430c0 2024-03-25T18:16:17.361 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.360+0000 7fd278ff9640 1 -- 172.21.15.156:0/2184885762 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fd274001c30 con 0x7fd27c1430c0 2024-03-25T18:16:17.361 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.360+0000 7fd278ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:17.361 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.360+0000 7fd278ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:17.361 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.360+0000 7fd278ff9640 10 monclient(hunting): dump: 2024-03-25T18:16:17.361 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:17.361 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:17.362 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:17.362 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:17.362 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:17.362 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:17.362 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:17.362 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:17.362 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:17.362 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:17.362 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.360+0000 7fd278ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:17.363 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.361+0000 7fd278ff9640 1 -- 172.21.15.156:0/2184885762 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2745780618 0 0) 0x7fd27c1d76a0 con 0x7fd27c0a98f0 2024-03-25T18:16:17.363 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.361+0000 7fd278ff9640 1 -- 172.21.15.156:0/2184885762 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd268004eb0 con 0x7fd27c0a98f0 2024-03-25T18:16:17.363 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.361+0000 7fd278ff9640 1 -- 172.21.15.156:0/2184885762 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fd26c001b00 con 0x7fd27c0a98f0 2024-03-25T18:16:17.363 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.361+0000 7fd278ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:17.363 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.361+0000 7fd278ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:17.363 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.361+0000 7fd278ff9640 10 monclient(hunting): dump: 2024-03-25T18:16:17.363 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:17.363 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:17.363 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:17.363 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:17.363 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:17.363 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:17.364 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:17.364 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:17.364 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:17.364 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:17.372 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.370+0000 7fd278ff9640 1 -- 172.21.15.156:0/2184885762 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3266532107 0 0) 0x7fd27c1d82d0 con 0x7fd27c13f710 2024-03-25T18:16:17.372 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.371+0000 7fd278ff9640 1 -- 172.21.15.156:0/2184885762 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd27c1d76a0 con 0x7fd27c13f710 2024-03-25T18:16:17.372 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.371+0000 7fd278ff9640 1 -- 172.21.15.156:0/2184885762 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fd2700018b0 con 0x7fd27c13f710 2024-03-25T18:16:17.372 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.371+0000 7fd278ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:17.372 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.371+0000 7fd278ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:17.372 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.371+0000 7fd278ff9640 10 monclient(hunting): dump: 2024-03-25T18:16:17.372 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:17.372 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:17.372 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:17.372 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:17.372 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:17.373 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:17.373 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:17.373 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:17.373 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:17.373 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:17.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.380+0000 7fd278ff9640 1 -- 172.21.15.156:0/2184885762 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2743693769 0 0) 0x7fd27c0abb90 con 0x7fd27c1430c0 2024-03-25T18:16:17.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.380+0000 7fd278ff9640 1 -- 172.21.15.156:0/2184885762 >> v1:172.21.15.156:6790/0 conn(0x7fd27c13f710 legacy=0x7fd27c1d7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.380+0000 7fd278ff9640 1 -- 172.21.15.156:0/2184885762 >> v1:172.21.15.194:6789/0 conn(0x7fd27c0a98f0 legacy=0x7fd27c1385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.380+0000 7fd278ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:16:17.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.380+0000 7fd278ff9640 1 monclient: found mon.a 2024-03-25T18:16:17.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.380+0000 7fd278ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:17.382 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.380+0000 7fd278ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:16:17.382 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.380+0000 7fd278ff9640 1 -- 172.21.15.156:0/2184885762 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd27c134720 con 0x7fd27c1430c0 2024-03-25T18:16:17.382 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.380+0000 7fd278ff9640 10 monclient: _finish_auth 0 2024-03-25T18:16:17.382 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.380+0000 7fd278ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:16:17.382 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.380+0000 7fd278ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:17.382 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.380+0000 7fd282674640 5 monclient: authenticate success, global_id 4581 2024-03-25T18:16:17.382 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.380+0000 7fd282674640 10 monclient: _renew_subs 2024-03-25T18:16:17.382 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.380+0000 7fd282674640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:16:17.382 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.380+0000 7fd282674640 1 -- 172.21.15.156:0/2184885762 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd27c134970 con 0x7fd27c1430c0 2024-03-25T18:16:17.382 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.380+0000 7fd282674640 10 client.4581.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:17.382 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.380+0000 7fd282674640 10 monclient: _renew_subs 2024-03-25T18:16:17.382 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.380+0000 7fd282674640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:16:17.382 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.380+0000 7fd282674640 1 -- 172.21.15.156:0/2184885762 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd27c137a10 con 0x7fd27c1430c0 2024-03-25T18:16:17.382 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.380+0000 7fd282674640 1 librados: init done 2024-03-25T18:16:17.383 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.381+0000 7fd282674640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:17.383 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.381+0000 7fd282674640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:17.383 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.381+0000 7fd282674640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:16:17.383 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.381+0000 7fd282674640 1 -- 172.21.15.156:0/2184885762 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd27c074cc0 con 0x7fd27c1430c0 2024-03-25T18:16:17.715 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.713+0000 7fd278ff9640 1 -- 172.21.15.156:0/2184885762 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd274003990 con 0x7fd27c1430c0 2024-03-25T18:16:17.715 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.713+0000 7fd278ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:17.715 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.713+0000 7fd278ff9640 1 -- 172.21.15.156:0/2184885762 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fd274003da0 con 0x7fd27c1430c0 2024-03-25T18:16:17.715 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.713+0000 7fd278ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:17.715 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.713+0000 7fd278ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:17.715 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.713+0000 7fd278ff9640 10 monclient: dump: 2024-03-25T18:16:17.715 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:17.715 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:17.715 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:17.715 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:17.715 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:17.715 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:17.716 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:17.716 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:17.716 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:17.716 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:17.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.713+0000 7fd27a7fc640 4 set_mon_vals no callback set 2024-03-25T18:16:17.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.714+0000 7fd278ff9640 1 -- 172.21.15.156:0/2184885762 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7fd274003fc0 con 0x7fd27c1430c0 2024-03-25T18:16:17.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.714+0000 7fd27bfff640 10 monclient: get_auth_request con 0x7fd26806e0f0 auth_method 0 2024-03-25T18:16:17.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.715+0000 7fd278ff9640 10 client.4581.objecter ms_handle_connect 0x7fd26806e0f0 2024-03-25T18:16:17.721 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.720+0000 7fd278ff9640 1 -- 172.21.15.156:0/2184885762 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7fd274054a90 con 0x7fd27c1430c0 2024-03-25T18:16:17.721 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.720+0000 7fd278ff9640 10 client.4581.objecter ms_dispatch 0x7fd27c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:17.721 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.720+0000 7fd278ff9640 3 client.4581.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:17.721 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.720+0000 7fd278ff9640 3 client.4581.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:17.721 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.720+0000 7fd278ff9640 20 client.4581.objecter dump_active .. 0 homeless 2024-03-25T18:16:17.721 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.720+0000 7fd278ff9640 1 -- 172.21.15.156:0/2184885762 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd2740505a0 con 0x7fd27c1430c0 2024-03-25T18:16:17.721 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.720+0000 7fd278ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:17.722 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.720+0000 7fd278ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:17.972 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.971+0000 7fd282674640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:17.972 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.971+0000 7fd282674640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:17.972 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.971+0000 7fd282674640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:16:17.972 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:17.971+0000 7fd282674640 1 -- 172.21.15.156:0/2184885762 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd27c1d7830 con 0x7fd27c1430c0 2024-03-25T18:16:18.029 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:18.028+0000 7fd278ff9640 1 -- 172.21.15.156:0/2184885762 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7fd2740524d0 con 0x7fd27c1430c0 2024-03-25T18:16:18.029 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:18.028+0000 7fd278ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:18.029 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:18.028+0000 7fd278ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:18.029 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:16:18.029 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:18.032 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:18.030+0000 7fd282674640 10 librados: watch_flush enter 2024-03-25T18:16:18.032 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:18.030+0000 7fd282674640 10 librados: watch_flush exit 2024-03-25T18:16:18.032 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:18.030+0000 7fd282674640 20 client.4581.objecter shutdown clearing up homeless session... 2024-03-25T18:16:18.032 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:18.030+0000 7fd282674640 10 client.4581.objecter successfully canceled tick 2024-03-25T18:16:18.032 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:18.030+0000 7fd282674640 1 -- 172.21.15.156:0/2184885762 >> v1:172.21.15.194:6800/2946427713 conn(0x7fd26806e0f0 legacy=0x7fd2680705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:18.032 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:18.030+0000 7fd282674640 10 monclient: shutdown 2024-03-25T18:16:18.032 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:18.030+0000 7fd282674640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:18.032 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:18.030+0000 7fd282674640 1 -- 172.21.15.156:0/2184885762 >> v1:172.21.15.156:6789/0 conn(0x7fd27c1430c0 legacy=0x7fd27c134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:18.033 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:18.030+0000 7fd282674640 1 -- 172.21.15.156:0/2184885762 shutdown_connections 2024-03-25T18:16:18.033 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:18.030+0000 7fd282674640 1 -- 172.21.15.156:0/2184885762 >> 172.21.15.156:0/2184885762 conn(0x7fd27c0b7ea0 msgr2=0x7fd27c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:18.033 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:18.030+0000 7fd282674640 1 -- 172.21.15.156:0/2184885762 shutdown_connections 2024-03-25T18:16:18.033 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:18.031+0000 7fd282674640 1 -- 172.21.15.156:0/2184885762 wait complete. 2024-03-25T18:16:18.033 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:18.031+0000 7fd282674640 1 librados: shutdown 2024-03-25T18:16:18.045 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:19.848 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:08:25.839 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:19.895 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:08:25.885 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:19.926 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:08:25.916 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:19.955 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:08:25.945 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:19.962 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:08:25.952 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:19.962 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:08:25.928 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:19.966 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:08:25.933 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:19.997 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:08:25.987 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:20.005 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:08:25.972 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:24.852 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:24.967 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.966+0000 7fc5b73e2640 10 monclient: get_monmap_and_config 2024-03-25T18:16:24.968 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.966+0000 7fc5b73e2640 10 monclient: build_initial_monmap 2024-03-25T18:16:24.968 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.966+0000 7fc5b73e2640 10 monclient: monmap: 2024-03-25T18:16:24.968 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:16:24.968 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:24.968 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:16:24.967815+0000 2024-03-25T18:16:24.968 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:16:24.967815+0000 2024-03-25T18:16:24.968 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:24.968 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:24.968 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:16:24.968 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:16:24.968 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:16:24.969 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:24.970 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.967+0000 7fc5b73e2640 1 Processor -- start 2024-03-25T18:16:24.970 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.967+0000 7fc5b73e2640 1 -- start start 2024-03-25T18:16:24.970 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.967+0000 7fc5b73e2640 10 monclient: init 2024-03-25T18:16:24.970 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.968+0000 7fc5b73e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:24.970 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.968+0000 7fc5b73e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:24.970 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.968+0000 7fc5b73e2640 10 monclient(hunting): picked mon.noname-c con 0x7fc5b00b4d60 addr v1:172.21.15.194:6789/0 2024-03-25T18:16:24.970 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.968+0000 7fc5b73e2640 10 monclient(hunting): picked mon.noname-b con 0x7fc5b00b1730 addr v1:172.21.15.156:6790/0 2024-03-25T18:16:24.970 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.968+0000 7fc5b73e2640 10 monclient(hunting): picked mon.noname-a con 0x7fc5b014be90 addr v1:172.21.15.156:6789/0 2024-03-25T18:16:24.970 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.968+0000 7fc5b73e2640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b00af940 con 0x7fc5b014be90 2024-03-25T18:16:24.970 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.968+0000 7fc5b73e2640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b00b0a70 con 0x7fc5b00b4d60 2024-03-25T18:16:24.970 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.968+0000 7fc5b73e2640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b00b0d50 con 0x7fc5b00b1730 2024-03-25T18:16:24.971 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.968+0000 7fc5b73e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:24.971 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.968+0000 7fc5b73e2640 10 monclient(hunting): authenticate will time out at 7466.279342s 2024-03-25T18:16:24.975 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.974+0000 7fc5b517e640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fc5b00b4d60 0x7fc5b013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:55660/0 (socket says 172.21.15.156:55660) 2024-03-25T18:16:24.976 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.975+0000 7fc5b517e640 1 -- 172.21.15.156:0/353519246 learned_addr learned my addr 172.21.15.156:0/353519246 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:16:24.986 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.985+0000 7fc5a7fff640 1 -- 172.21.15.156:0/353519246 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700586724 0 0) 0x7fc5b00b0a70 con 0x7fc5b00b4d60 2024-03-25T18:16:24.986 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.985+0000 7fc5a7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:24.986 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.985+0000 7fc5a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:24.987 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.985+0000 7fc5a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:24.987 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.985+0000 7fc5a7fff640 10 monclient(hunting): my global_id is 4591 2024-03-25T18:16:24.987 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.985+0000 7fc5a7fff640 1 -- 172.21.15.156:0/353519246 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc59c003520 con 0x7fc5b00b4d60 2024-03-25T18:16:24.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.994+0000 7fc5a7fff640 1 -- 172.21.15.156:0/353519246 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1632644822 0 0) 0x7fc59c003520 con 0x7fc5b00b4d60 2024-03-25T18:16:24.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.994+0000 7fc5a7fff640 1 -- 172.21.15.156:0/353519246 >> v1:172.21.15.156:6790/0 conn(0x7fc5b00b1730 legacy=0x7fc5b00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:24.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.994+0000 7fc5a7fff640 1 -- 172.21.15.156:0/353519246 >> v1:172.21.15.156:6789/0 conn(0x7fc5b014be90 legacy=0x7fc5b014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:24.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.994+0000 7fc5a7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:24.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.994+0000 7fc5a7fff640 1 monclient: found mon.noname-c 2024-03-25T18:16:24.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.994+0000 7fc5a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:24.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.994+0000 7fc5a7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:16:24.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.994+0000 7fc5a7fff640 1 -- 172.21.15.156:0/353519246 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b00b1030 con 0x7fc5b00b4d60 2024-03-25T18:16:24.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.994+0000 7fc5a7fff640 10 monclient: _finish_auth 0 2024-03-25T18:16:24.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.994+0000 7fc5a7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:24.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.994+0000 7fc5a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:24.997 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.994+0000 7fc5a7fff640 1 -- 172.21.15.156:0/353519246 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc5ac001690 con 0x7fc5b00b4d60 2024-03-25T18:16:24.997 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.994+0000 7fc5a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:24.997 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.994+0000 7fc5a7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:24.997 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.994+0000 7fc5a7fff640 10 monclient: dump: 2024-03-25T18:16:24.997 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:24.997 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:24.997 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:24.997 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:24.997 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:24.997 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:24.997 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:24.997 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:24.997 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:24.997 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:24.998 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.994+0000 7fc5b73e2640 5 monclient: authenticate success, global_id 4591 2024-03-25T18:16:24.998 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:24.994+0000 7fc5b73e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:25.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.003+0000 7fc5a7fff640 1 -- 172.21.15.156:0/353519246 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5ac0017f0 con 0x7fc5b00b4d60 2024-03-25T18:16:25.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.003+0000 7fc5a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:25.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.003+0000 7fc5a7fff640 1 -- 172.21.15.156:0/353519246 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc5ac001c00 con 0x7fc5b00b4d60 2024-03-25T18:16:25.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.003+0000 7fc5b73e2640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:25.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.003+0000 7fc5b73e2640 4 set_mon_vals no callback set 2024-03-25T18:16:25.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.003+0000 7fc5a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:25.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.003+0000 7fc5a7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:25.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.003+0000 7fc5a7fff640 10 monclient: dump: 2024-03-25T18:16:25.004 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:25.004 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:25.005 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:25.005 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:25.005 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:25.005 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:25.005 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:25.005 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:25.005 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:25.005 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:25.006 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.003+0000 7fc5b73e2640 10 monclient: shutdown 2024-03-25T18:16:25.006 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.003+0000 7fc5b73e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:25.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.003+0000 7fc5b73e2640 1 -- 172.21.15.156:0/353519246 >> v1:172.21.15.194:6789/0 conn(0x7fc5b00b4d60 legacy=0x7fc5b013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:25.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.004+0000 7fc5b73e2640 1 -- 172.21.15.156:0/353519246 shutdown_connections 2024-03-25T18:16:25.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.004+0000 7fc5b73e2640 1 -- 172.21.15.156:0/353519246 >> 172.21.15.156:0/353519246 conn(0x7fc5b00617a0 msgr2=0x7fc5b0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:25.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.004+0000 7fc5b73e2640 1 -- 172.21.15.156:0/353519246 shutdown_connections 2024-03-25T18:16:25.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.004+0000 7fc5b73e2640 1 -- 172.21.15.156:0/353519246 wait complete. 2024-03-25T18:16:25.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.004+0000 7fc5b73e2640 10 monclient: build_initial_monmap 2024-03-25T18:16:25.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.004+0000 7fc5b73e2640 10 monclient: monmap: 2024-03-25T18:16:25.007 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:16:25.007 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:25.007 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:16:25.007 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:16:25.007 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:25.007 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:16:25.007 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:16:25.008 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:16:25.008 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:16:25.008 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:25.008 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.005+0000 7fc5b73e2640 1 librados: starting msgr at 2024-03-25T18:16:25.008 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.005+0000 7fc5b73e2640 1 librados: starting objecter 2024-03-25T18:16:25.008 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.005+0000 7fc5b73e2640 1 Processor -- start 2024-03-25T18:16:25.008 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.005+0000 7fc5b73e2640 1 -- start start 2024-03-25T18:16:25.008 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.005+0000 7fc5b73e2640 1 librados: setting wanted keys 2024-03-25T18:16:25.008 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.005+0000 7fc5b73e2640 1 librados: calling monclient init 2024-03-25T18:16:25.008 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.005+0000 7fc5b73e2640 10 monclient: init 2024-03-25T18:16:25.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.006+0000 7fc5b73e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:25.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.006+0000 7fc5b73e2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:25.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.006+0000 7fc5b73e2640 10 monclient(hunting): picked mon.noname-b con 0x7fc5b00b1730 addr v1:172.21.15.194:6789/0 2024-03-25T18:16:25.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.006+0000 7fc5b73e2640 10 monclient(hunting): picked mon.noname-a con 0x7fc5b00b4d60 addr v1:172.21.15.156:6789/0 2024-03-25T18:16:25.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.006+0000 7fc5b73e2640 10 monclient(hunting): picked mon.noname-c con 0x7fc5b014be90 addr v1:172.21.15.156:6790/0 2024-03-25T18:16:25.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.006+0000 7fc5b73e2640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b0140d00 con 0x7fc5b00b4d60 2024-03-25T18:16:25.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.006+0000 7fc5b73e2640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b0141cc0 con 0x7fc5b00b1730 2024-03-25T18:16:25.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.006+0000 7fc5b73e2640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b01d8470 con 0x7fc5b014be90 2024-03-25T18:16:25.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.006+0000 7fc5b73e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:25.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.006+0000 7fc5b73e2640 10 monclient(hunting): authenticate will time out at 7466.317352s 2024-03-25T18:16:25.016 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.015+0000 7fc5b497d640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fc5b00b4d60 0x7fc5b01d7d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:36680/0 (socket says 172.21.15.156:36680) 2024-03-25T18:16:25.016 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.015+0000 7fc5b497d640 1 -- 172.21.15.156:0/43744013 learned_addr learned my addr 172.21.15.156:0/43744013 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:16:25.019 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.018+0000 7fc5a5ffb640 10 client.?.objecter ms_handle_connect 0x7fc5b014be90 2024-03-25T18:16:25.019 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.018+0000 7fc5a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:25.023 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.023+0000 7fc5a5ffb640 10 client.?.objecter ms_handle_connect 0x7fc5b00b4d60 2024-03-25T18:16:25.024 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.023+0000 7fc5a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:25.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.026+0000 7fc5a5ffb640 10 client.?.objecter ms_handle_connect 0x7fc5b00b1730 2024-03-25T18:16:25.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.026+0000 7fc5a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:25.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.036+0000 7fc5a5ffb640 1 -- 172.21.15.156:0/43744013 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021887691 0 0) 0x7fc5b0140d00 con 0x7fc5b00b4d60 2024-03-25T18:16:25.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.036+0000 7fc5a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:25.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.036+0000 7fc5a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:25.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.036+0000 7fc5a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:25.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.036+0000 7fc5a5ffb640 10 monclient(hunting): my global_id is 4587 2024-03-25T18:16:25.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.036+0000 7fc5a5ffb640 1 -- 172.21.15.156:0/43744013 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc594004eb0 con 0x7fc5b00b4d60 2024-03-25T18:16:25.041 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.040+0000 7fc5a5ffb640 1 -- 172.21.15.156:0/43744013 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 76809356 0 0) 0x7fc5b01d8470 con 0x7fc5b014be90 2024-03-25T18:16:25.041 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.040+0000 7fc5a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:25.041 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.040+0000 7fc5a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:25.041 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.040+0000 7fc5a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:25.041 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.040+0000 7fc5a5ffb640 10 monclient(hunting): my global_id is 4598 2024-03-25T18:16:25.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.040+0000 7fc5a5ffb640 1 -- 172.21.15.156:0/43744013 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b0140d00 con 0x7fc5b014be90 2024-03-25T18:16:25.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.042+0000 7fc5a5ffb640 1 -- 172.21.15.156:0/43744013 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1956485825 0 0) 0x7fc5b0141cc0 con 0x7fc5b00b1730 2024-03-25T18:16:25.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.042+0000 7fc5a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:25.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.042+0000 7fc5a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:25.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.042+0000 7fc5a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:25.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.042+0000 7fc5a5ffb640 10 monclient(hunting): my global_id is 4594 2024-03-25T18:16:25.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.042+0000 7fc5a5ffb640 1 -- 172.21.15.156:0/43744013 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b01d8470 con 0x7fc5b00b1730 2024-03-25T18:16:25.049 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.048+0000 7fc5a5ffb640 1 -- 172.21.15.156:0/43744013 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2044735673 0 0) 0x7fc594004eb0 con 0x7fc5b00b4d60 2024-03-25T18:16:25.049 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.048+0000 7fc5a5ffb640 1 -- 172.21.15.156:0/43744013 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5b0141cc0 con 0x7fc5b00b4d60 2024-03-25T18:16:25.049 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.048+0000 7fc5a5ffb640 1 -- 172.21.15.156:0/43744013 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc5a00019d0 con 0x7fc5b00b4d60 2024-03-25T18:16:25.050 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.048+0000 7fc5a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:25.050 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.048+0000 7fc5a5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:25.050 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.048+0000 7fc5a5ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:25.050 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:25.050 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:25.050 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:25.050 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:25.050 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:25.050 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:25.050 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:25.050 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:25.050 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:25.050 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:25.050 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.048+0000 7fc5a5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:25.058 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.056+0000 7fc5a5ffb640 1 -- 172.21.15.156:0/43744013 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2283408816 0 0) 0x7fc5b0140d00 con 0x7fc5b014be90 2024-03-25T18:16:25.058 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.056+0000 7fc5a5ffb640 1 -- 172.21.15.156:0/43744013 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc594004eb0 con 0x7fc5b014be90 2024-03-25T18:16:25.058 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.057+0000 7fc5a5ffb640 1 -- 172.21.15.156:0/43744013 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc5a8001b10 con 0x7fc5b014be90 2024-03-25T18:16:25.058 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.057+0000 7fc5a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:25.058 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.057+0000 7fc5a5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:25.058 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.057+0000 7fc5a5ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:25.058 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:25.058 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:25.058 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:25.058 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:25.058 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:25.059 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:25.059 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:25.059 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:25.059 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:25.059 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:25.063 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.062+0000 7fc5a5ffb640 1 -- 172.21.15.156:0/43744013 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1780118712 0 0) 0x7fc5b01d8470 con 0x7fc5b00b1730 2024-03-25T18:16:25.063 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.062+0000 7fc5a5ffb640 1 -- 172.21.15.156:0/43744013 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5b0140d00 con 0x7fc5b00b1730 2024-03-25T18:16:25.064 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.062+0000 7fc5a5ffb640 1 -- 172.21.15.156:0/43744013 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc5ac001b00 con 0x7fc5b00b1730 2024-03-25T18:16:25.064 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.062+0000 7fc5a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:25.064 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.062+0000 7fc5a5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:25.064 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.062+0000 7fc5a5ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:25.064 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:25.064 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:25.064 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:25.064 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:25.064 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:25.064 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:25.064 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:25.064 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:25.064 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:25.065 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:25.066 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.063+0000 7fc5a5ffb640 1 -- 172.21.15.156:0/43744013 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1102957558 0 0) 0x7fc5b0141cc0 con 0x7fc5b00b4d60 2024-03-25T18:16:25.066 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.063+0000 7fc5a5ffb640 1 -- 172.21.15.156:0/43744013 >> v1:172.21.15.156:6790/0 conn(0x7fc5b014be90 legacy=0x7fc5b0134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:25.066 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.063+0000 7fc5a5ffb640 1 -- 172.21.15.156:0/43744013 >> v1:172.21.15.194:6789/0 conn(0x7fc5b00b1730 legacy=0x7fc5b01405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:25.066 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.063+0000 7fc5a5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:25.066 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.063+0000 7fc5a5ffb640 1 monclient: found mon.a 2024-03-25T18:16:25.066 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.063+0000 7fc5a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:25.067 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.063+0000 7fc5a5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:16:25.067 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.063+0000 7fc5a5ffb640 1 -- 172.21.15.156:0/43744013 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b01d8600 con 0x7fc5b00b4d60 2024-03-25T18:16:25.067 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.063+0000 7fc5a5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:25.067 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.063+0000 7fc5a5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:25.067 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.063+0000 7fc5a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:25.067 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.063+0000 7fc5b73e2640 5 monclient: authenticate success, global_id 4587 2024-03-25T18:16:25.067 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.063+0000 7fc5b73e2640 10 monclient: _renew_subs 2024-03-25T18:16:25.067 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.063+0000 7fc5b73e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:16:25.067 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.063+0000 7fc5b73e2640 1 -- 172.21.15.156:0/43744013 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b013b3c0 con 0x7fc5b00b4d60 2024-03-25T18:16:25.067 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.063+0000 7fc5b73e2640 10 client.4587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:25.067 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.063+0000 7fc5b73e2640 10 monclient: _renew_subs 2024-03-25T18:16:25.067 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.063+0000 7fc5b73e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:16:25.067 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.063+0000 7fc5b73e2640 1 -- 172.21.15.156:0/43744013 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b01425a0 con 0x7fc5b00b4d60 2024-03-25T18:16:25.067 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.063+0000 7fc5b73e2640 1 librados: init done 2024-03-25T18:16:25.067 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.064+0000 7fc5b73e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:25.068 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.064+0000 7fc5b73e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:25.068 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.064+0000 7fc5b73e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:16:25.068 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.064+0000 7fc5b73e2640 1 -- 172.21.15.156:0/43744013 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b00b2270 con 0x7fc5b00b4d60 2024-03-25T18:16:25.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.384+0000 7fc5a5ffb640 1 -- 172.21.15.156:0/43744013 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5a0003610 con 0x7fc5b00b4d60 2024-03-25T18:16:25.385 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.384+0000 7fc5a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:25.386 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.384+0000 7fc5a5ffb640 1 -- 172.21.15.156:0/43744013 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc5a0003a20 con 0x7fc5b00b4d60 2024-03-25T18:16:25.386 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.384+0000 7fc5a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:25.386 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.384+0000 7fc5a5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:25.386 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.384+0000 7fc5a5ffb640 10 monclient: dump: 2024-03-25T18:16:25.386 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:25.386 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:25.386 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:25.386 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:25.386 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:25.386 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:25.386 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:25.386 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:25.387 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:25.387 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:25.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.384+0000 7fc5a77fe640 4 set_mon_vals no callback set 2024-03-25T18:16:25.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.384+0000 7fc5a5ffb640 1 -- 172.21.15.156:0/43744013 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7fc5a0003c40 con 0x7fc5b00b4d60 2024-03-25T18:16:25.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.385+0000 7fc5b517e640 10 monclient: get_auth_request con 0x7fc59406e0f0 auth_method 0 2024-03-25T18:16:25.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.385+0000 7fc5a5ffb640 10 client.4587.objecter ms_handle_connect 0x7fc59406e0f0 2024-03-25T18:16:25.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.390+0000 7fc5a5ffb640 1 -- 172.21.15.156:0/43744013 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7fc5a00547d0 con 0x7fc5b00b4d60 2024-03-25T18:16:25.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.390+0000 7fc5a5ffb640 10 client.4587.objecter ms_dispatch 0x7fc5b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:25.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.390+0000 7fc5a5ffb640 3 client.4587.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:25.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.390+0000 7fc5a5ffb640 3 client.4587.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:25.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.390+0000 7fc5a5ffb640 20 client.4587.objecter dump_active .. 0 homeless 2024-03-25T18:16:25.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.390+0000 7fc5a5ffb640 1 -- 172.21.15.156:0/43744013 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc5a00502e0 con 0x7fc5b00b4d60 2024-03-25T18:16:25.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.390+0000 7fc5a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:25.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.390+0000 7fc5a5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:25.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.608+0000 7fc5b73e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:25.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.608+0000 7fc5b73e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:25.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.608+0000 7fc5b73e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:16:25.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.608+0000 7fc5b73e2640 1 -- 172.21.15.156:0/43744013 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5b00627b0 con 0x7fc5b00b4d60 2024-03-25T18:16:25.634 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.633+0000 7fc5a5ffb640 1 -- 172.21.15.156:0/43744013 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7fc5a0052210 con 0x7fc5b00b4d60 2024-03-25T18:16:25.634 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.633+0000 7fc5a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:25.634 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.633+0000 7fc5a5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:25.635 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:16:25.635 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:25.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.636+0000 7fc5b73e2640 10 librados: watch_flush enter 2024-03-25T18:16:25.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.636+0000 7fc5b73e2640 10 librados: watch_flush exit 2024-03-25T18:16:25.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.636+0000 7fc5b73e2640 20 client.4587.objecter shutdown clearing up homeless session... 2024-03-25T18:16:25.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.636+0000 7fc5b73e2640 10 client.4587.objecter successfully canceled tick 2024-03-25T18:16:25.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.636+0000 7fc5b73e2640 1 -- 172.21.15.156:0/43744013 >> v1:172.21.15.194:6800/2946427713 conn(0x7fc59406e0f0 legacy=0x7fc5940705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:25.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.636+0000 7fc5b73e2640 10 monclient: shutdown 2024-03-25T18:16:25.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.636+0000 7fc5b73e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:25.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.636+0000 7fc5b73e2640 1 -- 172.21.15.156:0/43744013 >> v1:172.21.15.156:6789/0 conn(0x7fc5b00b4d60 legacy=0x7fc5b01d7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:25.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.636+0000 7fc5b73e2640 1 -- 172.21.15.156:0/43744013 shutdown_connections 2024-03-25T18:16:25.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.636+0000 7fc5b73e2640 1 -- 172.21.15.156:0/43744013 >> 172.21.15.156:0/43744013 conn(0x7fc5b00617a0 msgr2=0x7fc5b00b6f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:25.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.636+0000 7fc5b73e2640 1 -- 172.21.15.156:0/43744013 shutdown_connections 2024-03-25T18:16:25.638 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.636+0000 7fc5b73e2640 1 -- 172.21.15.156:0/43744013 wait complete. 2024-03-25T18:16:25.639 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:25.636+0000 7fc5b73e2640 1 librados: shutdown 2024-03-25T18:16:25.649 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:32.456 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:32.571 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.570+0000 7fdc236fb640 10 monclient: get_monmap_and_config 2024-03-25T18:16:32.572 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.570+0000 7fdc236fb640 10 monclient: build_initial_monmap 2024-03-25T18:16:32.572 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.570+0000 7fdc236fb640 10 monclient: monmap: 2024-03-25T18:16:32.572 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:16:32.572 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:32.572 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:16:32.571900+0000 2024-03-25T18:16:32.572 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:16:32.571900+0000 2024-03-25T18:16:32.572 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:32.572 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:32.572 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:16:32.572 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:16:32.572 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:16:32.572 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:32.574 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.571+0000 7fdc236fb640 1 Processor -- start 2024-03-25T18:16:32.574 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.571+0000 7fdc236fb640 1 -- start start 2024-03-25T18:16:32.574 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.571+0000 7fdc236fb640 10 monclient: init 2024-03-25T18:16:32.574 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.572+0000 7fdc236fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:32.574 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.572+0000 7fdc236fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:32.574 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.572+0000 7fdc236fb640 10 monclient(hunting): picked mon.noname-b con 0x7fdc1c0a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:16:32.574 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.572+0000 7fdc236fb640 10 monclient(hunting): picked mon.noname-a con 0x7fdc1c13f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:16:32.574 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.572+0000 7fdc236fb640 10 monclient(hunting): picked mon.noname-c con 0x7fdc1c1430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:16:32.574 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.572+0000 7fdc236fb640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc1c142520 con 0x7fdc1c13f710 2024-03-25T18:16:32.574 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.572+0000 7fdc236fb640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc1c0b28a0 con 0x7fdc1c1430c0 2024-03-25T18:16:32.574 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.572+0000 7fdc236fb640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc1c0b2a30 con 0x7fdc1c0a98f0 2024-03-25T18:16:32.574 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.572+0000 7fdc236fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:32.575 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.572+0000 7fdc236fb640 10 monclient(hunting): authenticate will time out at 7473.883603s 2024-03-25T18:16:32.591 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.590+0000 7fdc21c98640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fdc1c1430c0 0x7fdc1c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:51992/0 (socket says 172.21.15.156:51992) 2024-03-25T18:16:32.591 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.590+0000 7fdc21c98640 1 -- 172.21.15.156:0/2582384086 learned_addr learned my addr 172.21.15.156:0/2582384086 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:16:32.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.606+0000 7fdc13fff640 1 -- 172.21.15.156:0/2582384086 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2839494533 0 0) 0x7fdc1c142520 con 0x7fdc1c13f710 2024-03-25T18:16:32.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.606+0000 7fdc13fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.606+0000 7fdc13fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.606+0000 7fdc13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.606+0000 7fdc13fff640 10 monclient(hunting): my global_id is 4590 2024-03-25T18:16:32.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.606+0000 7fdc13fff640 1 -- 172.21.15.156:0/2582384086 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc08003520 con 0x7fdc1c13f710 2024-03-25T18:16:32.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.607+0000 7fdc13fff640 1 -- 172.21.15.156:0/2582384086 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 410816311 0 0) 0x7fdc1c0b28a0 con 0x7fdc1c1430c0 2024-03-25T18:16:32.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.607+0000 7fdc13fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.607+0000 7fdc13fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.607+0000 7fdc13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.607+0000 7fdc13fff640 10 monclient(hunting): my global_id is 4597 2024-03-25T18:16:32.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.607+0000 7fdc13fff640 1 -- 172.21.15.156:0/2582384086 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc1c142520 con 0x7fdc1c1430c0 2024-03-25T18:16:32.621 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.619+0000 7fdc13fff640 1 -- 172.21.15.156:0/2582384086 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422791873 0 0) 0x7fdc1c0b2a30 con 0x7fdc1c0a98f0 2024-03-25T18:16:32.621 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.619+0000 7fdc13fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.621 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.619+0000 7fdc13fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.621 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.619+0000 7fdc13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.621 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.619+0000 7fdc13fff640 10 monclient(hunting): my global_id is 4601 2024-03-25T18:16:32.621 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.619+0000 7fdc13fff640 1 -- 172.21.15.156:0/2582384086 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc1c0b28a0 con 0x7fdc1c0a98f0 2024-03-25T18:16:32.621 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.619+0000 7fdc13fff640 1 -- 172.21.15.156:0/2582384086 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 876110749 0 0) 0x7fdc08003520 con 0x7fdc1c13f710 2024-03-25T18:16:32.621 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.619+0000 7fdc13fff640 1 -- 172.21.15.156:0/2582384086 >> v1:172.21.15.156:6790/0 conn(0x7fdc1c0a98f0 legacy=0x7fdc1c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.621 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.619+0000 7fdc13fff640 1 -- 172.21.15.156:0/2582384086 >> v1:172.21.15.194:6789/0 conn(0x7fdc1c1430c0 legacy=0x7fdc1c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.621 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.619+0000 7fdc13fff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:32.622 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.619+0000 7fdc13fff640 1 monclient: found mon.noname-a 2024-03-25T18:16:32.622 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.619+0000 7fdc13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:32.622 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.619+0000 7fdc13fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:16:32.622 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.619+0000 7fdc13fff640 1 -- 172.21.15.156:0/2582384086 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1c0b44c0 con 0x7fdc1c13f710 2024-03-25T18:16:32.622 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.619+0000 7fdc13fff640 10 monclient: _finish_auth 0 2024-03-25T18:16:32.622 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.619+0000 7fdc13fff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:32.622 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.619+0000 7fdc13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:32.622 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.619+0000 7fdc13fff640 1 -- 172.21.15.156:0/2582384086 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fdc0c001610 con 0x7fdc1c13f710 2024-03-25T18:16:32.622 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.619+0000 7fdc13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.622 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.619+0000 7fdc13fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:32.622 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.619+0000 7fdc13fff640 10 monclient: dump: 2024-03-25T18:16:32.622 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:32.622 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:32.623 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:32.623 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:32.623 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.623 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:32.623 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:32.623 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:32.623 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:32.623 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:32.623 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.619+0000 7fdc236fb640 5 monclient: authenticate success, global_id 4590 2024-03-25T18:16:32.623 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.619+0000 7fdc236fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:32.629 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.628+0000 7fdc13fff640 1 -- 172.21.15.156:0/2582384086 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc0c001770 con 0x7fdc1c13f710 2024-03-25T18:16:32.629 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.628+0000 7fdc13fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:32.629 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.628+0000 7fdc13fff640 1 -- 172.21.15.156:0/2582384086 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fdc0c001bc0 con 0x7fdc1c13f710 2024-03-25T18:16:32.629 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.628+0000 7fdc13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.628+0000 7fdc13fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:32.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.628+0000 7fdc13fff640 10 monclient: dump: 2024-03-25T18:16:32.630 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:32.630 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:32.630 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:32.630 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:32.630 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.630 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:32.630 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:32.630 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:32.630 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:32.630 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:32.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.628+0000 7fdc236fb640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:32.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.628+0000 7fdc236fb640 4 set_mon_vals no callback set 2024-03-25T18:16:32.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.629+0000 7fdc236fb640 10 monclient: shutdown 2024-03-25T18:16:32.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.629+0000 7fdc236fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:32.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.629+0000 7fdc236fb640 1 -- 172.21.15.156:0/2582384086 >> v1:172.21.15.156:6789/0 conn(0x7fdc1c13f710 legacy=0x7fdc1c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.629+0000 7fdc236fb640 1 -- 172.21.15.156:0/2582384086 shutdown_connections 2024-03-25T18:16:32.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.629+0000 7fdc236fb640 1 -- 172.21.15.156:0/2582384086 >> 172.21.15.156:0/2582384086 conn(0x7fdc1c0b7ea0 msgr2=0x7fdc1c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:32.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.629+0000 7fdc236fb640 1 -- 172.21.15.156:0/2582384086 shutdown_connections 2024-03-25T18:16:32.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.629+0000 7fdc236fb640 1 -- 172.21.15.156:0/2582384086 wait complete. 2024-03-25T18:16:32.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.630+0000 7fdc236fb640 10 monclient: build_initial_monmap 2024-03-25T18:16:32.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.630+0000 7fdc236fb640 10 monclient: monmap: 2024-03-25T18:16:32.632 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:16:32.633 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:32.633 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:16:32.633 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:16:32.633 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:32.633 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:16:32.633 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:16:32.633 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:16:32.633 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:16:32.633 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:32.633 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.630+0000 7fdc236fb640 1 librados: starting msgr at 2024-03-25T18:16:32.633 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.630+0000 7fdc236fb640 1 librados: starting objecter 2024-03-25T18:16:32.633 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.631+0000 7fdc236fb640 1 Processor -- start 2024-03-25T18:16:32.633 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.631+0000 7fdc236fb640 1 -- start start 2024-03-25T18:16:32.633 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.631+0000 7fdc236fb640 1 librados: setting wanted keys 2024-03-25T18:16:32.634 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.631+0000 7fdc236fb640 1 librados: calling monclient init 2024-03-25T18:16:32.634 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.631+0000 7fdc236fb640 10 monclient: init 2024-03-25T18:16:32.635 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.631+0000 7fdc236fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:32.635 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.631+0000 7fdc236fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:32.635 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.631+0000 7fdc236fb640 10 monclient(hunting): picked mon.noname-a con 0x7fdc1c0a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:16:32.635 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.631+0000 7fdc236fb640 10 monclient(hunting): picked mon.noname-b con 0x7fdc1c13f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:16:32.635 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.631+0000 7fdc236fb640 10 monclient(hunting): picked mon.noname-c con 0x7fdc1c1430c0 addr v1:172.21.15.156:6790/0 2024-03-25T18:16:32.635 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.632+0000 7fdc236fb640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc1c138e20 con 0x7fdc1c0a98f0 2024-03-25T18:16:32.635 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.632+0000 7fdc236fb640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc1c137ac0 con 0x7fdc1c13f710 2024-03-25T18:16:32.635 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.632+0000 7fdc236fb640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc1c137c50 con 0x7fdc1c1430c0 2024-03-25T18:16:32.635 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.632+0000 7fdc236fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:32.635 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.632+0000 7fdc236fb640 10 monclient(hunting): authenticate will time out at 7473.942944s 2024-03-25T18:16:32.644 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.643+0000 7fdc21c98640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7fdc1c1430c0 0x7fdc1c1e2d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:43796/0 (socket says 172.21.15.156:43796) 2024-03-25T18:16:32.644 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.643+0000 7fdc21c98640 1 -- 172.21.15.156:0/2242228790 learned_addr learned my addr 172.21.15.156:0/2242228790 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:16:32.645 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.644+0000 7fdc11ffb640 10 client.?.objecter ms_handle_connect 0x7fdc1c1430c0 2024-03-25T18:16:32.645 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.644+0000 7fdc11ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:32.648 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.647+0000 7fdc11ffb640 10 client.?.objecter ms_handle_connect 0x7fdc1c13f710 2024-03-25T18:16:32.648 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.647+0000 7fdc11ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:32.651 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.650+0000 7fdc11ffb640 10 client.?.objecter ms_handle_connect 0x7fdc1c0a98f0 2024-03-25T18:16:32.651 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.650+0000 7fdc11ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:32.659 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.658+0000 7fdc11ffb640 1 -- 172.21.15.156:0/2242228790 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307537724 0 0) 0x7fdc1c137ac0 con 0x7fdc1c13f710 2024-03-25T18:16:32.660 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.658+0000 7fdc11ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.660 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.658+0000 7fdc11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.660 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.658+0000 7fdc11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.660 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.658+0000 7fdc11ffb640 10 monclient(hunting): my global_id is 4600 2024-03-25T18:16:32.660 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.658+0000 7fdc11ffb640 1 -- 172.21.15.156:0/2242228790 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc00004eb0 con 0x7fdc1c13f710 2024-03-25T18:16:32.660 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.659+0000 7fdc11ffb640 1 -- 172.21.15.156:0/2242228790 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550415610 0 0) 0x7fdc1c137c50 con 0x7fdc1c1430c0 2024-03-25T18:16:32.660 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.659+0000 7fdc11ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.660 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.659+0000 7fdc11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.661 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.659+0000 7fdc11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.661 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.659+0000 7fdc11ffb640 10 monclient(hunting): my global_id is 4604 2024-03-25T18:16:32.661 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.659+0000 7fdc11ffb640 1 -- 172.21.15.156:0/2242228790 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc1c137ac0 con 0x7fdc1c1430c0 2024-03-25T18:16:32.676 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.675+0000 7fdc11ffb640 1 -- 172.21.15.156:0/2242228790 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 740158595 0 0) 0x7fdc1c138e20 con 0x7fdc1c0a98f0 2024-03-25T18:16:32.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.675+0000 7fdc11ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.675+0000 7fdc11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.675+0000 7fdc11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.675+0000 7fdc11ffb640 10 monclient(hunting): my global_id is 4593 2024-03-25T18:16:32.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.675+0000 7fdc11ffb640 1 -- 172.21.15.156:0/2242228790 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc1c137c50 con 0x7fdc1c0a98f0 2024-03-25T18:16:32.678 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.676+0000 7fdc11ffb640 1 -- 172.21.15.156:0/2242228790 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3113512772 0 0) 0x7fdc1c137ac0 con 0x7fdc1c1430c0 2024-03-25T18:16:32.678 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.676+0000 7fdc11ffb640 1 -- 172.21.15.156:0/2242228790 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc1c138e20 con 0x7fdc1c1430c0 2024-03-25T18:16:32.678 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.676+0000 7fdc11ffb640 1 -- 172.21.15.156:0/2242228790 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fdc14001c50 con 0x7fdc1c1430c0 2024-03-25T18:16:32.678 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.676+0000 7fdc11ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.678 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.676+0000 7fdc11ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:32.678 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.676+0000 7fdc11ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:32.678 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:32.678 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:32.678 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:32.678 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:32.678 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.679 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:32.679 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:32.679 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:32.679 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:32.679 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:32.679 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.676+0000 7fdc11ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:32.692 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.690+0000 7fdc11ffb640 1 -- 172.21.15.156:0/2242228790 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2265602763 0 0) 0x7fdc00004eb0 con 0x7fdc1c13f710 2024-03-25T18:16:32.692 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.690+0000 7fdc11ffb640 1 -- 172.21.15.156:0/2242228790 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc1c137ac0 con 0x7fdc1c13f710 2024-03-25T18:16:32.692 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.690+0000 7fdc11ffb640 1 -- 172.21.15.156:0/2242228790 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fdc0c001ac0 con 0x7fdc1c13f710 2024-03-25T18:16:32.692 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.690+0000 7fdc11ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.692 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.690+0000 7fdc11ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:32.692 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.690+0000 7fdc11ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:32.692 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:32.692 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:32.692 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:32.692 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:32.692 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.692 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:32.692 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:32.693 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:32.693 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:32.693 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:32.693 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.690+0000 7fdc11ffb640 1 -- 172.21.15.156:0/2242228790 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2120715146 0 0) 0x7fdc1c137c50 con 0x7fdc1c0a98f0 2024-03-25T18:16:32.693 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.690+0000 7fdc11ffb640 1 -- 172.21.15.156:0/2242228790 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc00004eb0 con 0x7fdc1c0a98f0 2024-03-25T18:16:32.693 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.690+0000 7fdc11ffb640 1 -- 172.21.15.156:0/2242228790 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fdc18001870 con 0x7fdc1c0a98f0 2024-03-25T18:16:32.693 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.690+0000 7fdc11ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.693 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.690+0000 7fdc11ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:32.693 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.690+0000 7fdc11ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:32.693 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:32.693 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:32.693 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:32.693 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:32.693 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.693 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:32.694 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:32.694 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:32.694 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:32.694 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:32.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.693+0000 7fdc11ffb640 1 -- 172.21.15.156:0/2242228790 <== mon.2 v1:172.21.15.156:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 107814196 0 0) 0x7fdc1c138e20 con 0x7fdc1c1430c0 2024-03-25T18:16:32.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.693+0000 7fdc11ffb640 1 -- 172.21.15.156:0/2242228790 >> v1:172.21.15.194:6789/0 conn(0x7fdc1c13f710 legacy=0x7fdc1c0b0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.693+0000 7fdc11ffb640 1 -- 172.21.15.156:0/2242228790 >> v1:172.21.15.156:6789/0 conn(0x7fdc1c0a98f0 legacy=0x7fdc1c1373b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.693+0000 7fdc11ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:32.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.693+0000 7fdc11ffb640 1 monclient: found mon.c 2024-03-25T18:16:32.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.693+0000 7fdc11ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:32.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.693+0000 7fdc11ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:16:32.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.693+0000 7fdc11ffb640 1 -- 172.21.15.156:0/2242228790 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1c1daa00 con 0x7fdc1c1430c0 2024-03-25T18:16:32.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.693+0000 7fdc11ffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:32.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.693+0000 7fdc11ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:32.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.693+0000 7fdc11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:32.696 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.693+0000 7fdc236fb640 5 monclient: authenticate success, global_id 4604 2024-03-25T18:16:32.696 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.693+0000 7fdc236fb640 10 monclient: _renew_subs 2024-03-25T18:16:32.696 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.693+0000 7fdc236fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:16:32.696 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.693+0000 7fdc236fb640 1 -- 172.21.15.156:0/2242228790 --> v1:172.21.15.156:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc1c136710 con 0x7fdc1c1430c0 2024-03-25T18:16:32.696 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.693+0000 7fdc236fb640 10 client.4604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:32.696 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.693+0000 7fdc236fb640 10 monclient: _renew_subs 2024-03-25T18:16:32.696 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.693+0000 7fdc236fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:16:32.696 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.693+0000 7fdc236fb640 1 -- 172.21.15.156:0/2242228790 --> v1:172.21.15.156:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc1c0b6e90 con 0x7fdc1c1430c0 2024-03-25T18:16:32.696 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.693+0000 7fdc236fb640 1 librados: init done 2024-03-25T18:16:32.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.694+0000 7fdc236fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:32.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.694+0000 7fdc236fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:32.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.694+0000 7fdc236fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:16:32.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.694+0000 7fdc236fb640 1 -- 172.21.15.156:0/2242228790 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc1c074cc0 con 0x7fdc1c1430c0 2024-03-25T18:16:32.993 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.992+0000 7fdc11ffb640 1 -- 172.21.15.156:0/2242228790 <== mon.2 v1:172.21.15.156:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc140020c0 con 0x7fdc1c1430c0 2024-03-25T18:16:32.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.992+0000 7fdc11ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:32.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.992+0000 7fdc11ffb640 1 -- 172.21.15.156:0/2242228790 <== mon.2 v1:172.21.15.156:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fdc14004d00 con 0x7fdc1c1430c0 2024-03-25T18:16:32.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.992+0000 7fdc11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.992+0000 7fdc11ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:32.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.992+0000 7fdc11ffb640 10 monclient: dump: 2024-03-25T18:16:32.994 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:32.994 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:32.994 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:32.994 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:32.994 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.994 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:32.994 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:32.995 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:32.995 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:32.995 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:32.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.992+0000 7fdc137fe640 4 set_mon_vals no callback set 2024-03-25T18:16:32.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.993+0000 7fdc11ffb640 1 -- 172.21.15.156:0/2242228790 <== mon.2 v1:172.21.15.156:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7fdc1401ad90 con 0x7fdc1c1430c0 2024-03-25T18:16:32.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.993+0000 7fdc21497640 10 monclient: get_auth_request con 0x7fdc0006e0f0 auth_method 0 2024-03-25T18:16:32.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.994+0000 7fdc11ffb640 10 client.4604.objecter ms_handle_connect 0x7fdc0006e0f0 2024-03-25T18:16:32.999 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.998+0000 7fdc11ffb640 1 -- 172.21.15.156:0/2242228790 <== mon.2 v1:172.21.15.156:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7fdc14054a30 con 0x7fdc1c1430c0 2024-03-25T18:16:32.999 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.998+0000 7fdc11ffb640 10 client.4604.objecter ms_dispatch 0x7fdc1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:33.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.998+0000 7fdc11ffb640 3 client.4604.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:33.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.998+0000 7fdc11ffb640 3 client.4604.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:33.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.998+0000 7fdc11ffb640 20 client.4604.objecter dump_active .. 0 homeless 2024-03-25T18:16:33.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.998+0000 7fdc11ffb640 1 -- 172.21.15.156:0/2242228790 <== mon.2 v1:172.21.15.156:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdc14050370 con 0x7fdc1c1430c0 2024-03-25T18:16:33.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.998+0000 7fdc11ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:33.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:32.998+0000 7fdc11ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:33.255 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:33.253+0000 7fdc236fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:33.255 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:33.253+0000 7fdc236fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:33.255 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:33.253+0000 7fdc236fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:16:33.255 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:33.253+0000 7fdc236fb640 1 -- 172.21.15.156:0/2242228790 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc1c1dac70 con 0x7fdc1c1430c0 2024-03-25T18:16:33.278 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:33.277+0000 7fdc11ffb640 1 -- 172.21.15.156:0/2242228790 <== mon.2 v1:172.21.15.156:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7fdc14052470 con 0x7fdc1c1430c0 2024-03-25T18:16:33.278 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:33.277+0000 7fdc11ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:33.278 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:33.277+0000 7fdc11ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:33.279 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:16:33.279 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:33.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:33.280+0000 7fdc236fb640 10 librados: watch_flush enter 2024-03-25T18:16:33.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:33.280+0000 7fdc236fb640 10 librados: watch_flush exit 2024-03-25T18:16:33.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:33.280+0000 7fdc236fb640 20 client.4604.objecter shutdown clearing up homeless session... 2024-03-25T18:16:33.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:33.280+0000 7fdc236fb640 10 client.4604.objecter successfully canceled tick 2024-03-25T18:16:33.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:33.281+0000 7fdc236fb640 1 -- 172.21.15.156:0/2242228790 >> v1:172.21.15.194:6800/2946427713 conn(0x7fdc0006e0f0 legacy=0x7fdc000705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:33.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:33.281+0000 7fdc236fb640 10 monclient: shutdown 2024-03-25T18:16:33.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:33.281+0000 7fdc236fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:33.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:33.281+0000 7fdc236fb640 1 -- 172.21.15.156:0/2242228790 >> v1:172.21.15.156:6790/0 conn(0x7fdc1c1430c0 legacy=0x7fdc1c1e2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:33.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:33.281+0000 7fdc236fb640 1 -- 172.21.15.156:0/2242228790 shutdown_connections 2024-03-25T18:16:33.283 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:33.281+0000 7fdc236fb640 1 -- 172.21.15.156:0/2242228790 >> 172.21.15.156:0/2242228790 conn(0x7fdc1c0b7ea0 msgr2=0x7fdc1c13c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:33.283 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:33.281+0000 7fdc236fb640 1 -- 172.21.15.156:0/2242228790 shutdown_connections 2024-03-25T18:16:33.283 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:33.281+0000 7fdc236fb640 1 -- 172.21.15.156:0/2242228790 wait complete. 2024-03-25T18:16:33.283 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:33.281+0000 7fdc236fb640 1 librados: shutdown 2024-03-25T18:16:33.294 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:39.820 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:08:45.810 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:39.825 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:08:45.815 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:39.858 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:08:45.849 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:39.867 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:08:45.834 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:39.880 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:08:45.847 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:39.883 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:08:45.873 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:39.893 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:08:45.883 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:39.911 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:08:45.877 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:39.924 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:08:45.914 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:40.100 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:40.215 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.213+0000 7f9d376eb640 10 monclient: get_monmap_and_config 2024-03-25T18:16:40.215 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.213+0000 7f9d376eb640 10 monclient: build_initial_monmap 2024-03-25T18:16:40.215 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.213+0000 7f9d376eb640 10 monclient: monmap: 2024-03-25T18:16:40.215 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:16:40.215 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:40.215 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:16:40.214882+0000 2024-03-25T18:16:40.215 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:16:40.214882+0000 2024-03-25T18:16:40.215 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:40.215 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:40.215 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:16:40.215 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:16:40.215 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:16:40.216 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:40.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.214+0000 7f9d376eb640 1 Processor -- start 2024-03-25T18:16:40.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.214+0000 7f9d376eb640 1 -- start start 2024-03-25T18:16:40.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.214+0000 7f9d376eb640 10 monclient: init 2024-03-25T18:16:40.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.215+0000 7f9d376eb640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:40.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.215+0000 7f9d376eb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:40.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.215+0000 7f9d376eb640 10 monclient(hunting): picked mon.noname-c con 0x7f9d300a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:16:40.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.215+0000 7f9d376eb640 10 monclient(hunting): picked mon.noname-b con 0x7f9d3013f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:16:40.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.215+0000 7f9d376eb640 10 monclient(hunting): picked mon.noname-a con 0x7f9d301430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:16:40.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.215+0000 7f9d376eb640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d30142520 con 0x7f9d301430c0 2024-03-25T18:16:40.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.215+0000 7f9d376eb640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d300b28a0 con 0x7f9d300a98f0 2024-03-25T18:16:40.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.215+0000 7f9d376eb640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d300b2a30 con 0x7f9d3013f710 2024-03-25T18:16:40.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.215+0000 7f9d376eb640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:40.218 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.215+0000 7f9d376eb640 10 monclient(hunting): authenticate will time out at 7481.526411s 2024-03-25T18:16:40.220 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.219+0000 7f9d34c86640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f9d3013f710 0x7f9d301429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:48502/0 (socket says 172.21.15.156:48502) 2024-03-25T18:16:40.220 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.219+0000 7f9d34c86640 1 -- 172.21.15.156:0/4264881786 learned_addr learned my addr 172.21.15.156:0/4264881786 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:16:40.233 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.232+0000 7f9d27fff640 1 -- 172.21.15.156:0/4264881786 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3857368790 0 0) 0x7f9d300b2a30 con 0x7f9d3013f710 2024-03-25T18:16:40.233 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.232+0000 7f9d27fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:40.233 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.232+0000 7f9d27fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:40.233 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.232+0000 7f9d27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:40.233 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.232+0000 7f9d27fff640 10 monclient(hunting): my global_id is 4607 2024-03-25T18:16:40.233 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.232+0000 7f9d27fff640 1 -- 172.21.15.156:0/4264881786 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d1c003520 con 0x7f9d3013f710 2024-03-25T18:16:40.239 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.238+0000 7f9d27fff640 1 -- 172.21.15.156:0/4264881786 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3180475702 0 0) 0x7f9d1c003520 con 0x7f9d3013f710 2024-03-25T18:16:40.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.238+0000 7f9d27fff640 1 -- 172.21.15.156:0/4264881786 >> v1:172.21.15.194:6789/0 conn(0x7f9d300a98f0 legacy=0x7f9d300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:40.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.238+0000 7f9d27fff640 1 -- 172.21.15.156:0/4264881786 >> v1:172.21.15.156:6789/0 conn(0x7f9d301430c0 legacy=0x7f9d3014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:40.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.238+0000 7f9d27fff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:40.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.238+0000 7f9d27fff640 1 monclient: found mon.noname-b 2024-03-25T18:16:40.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.238+0000 7f9d27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:40.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.238+0000 7f9d27fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.156:6790/0 2024-03-25T18:16:40.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.238+0000 7f9d27fff640 1 -- 172.21.15.156:0/4264881786 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d300b44c0 con 0x7f9d3013f710 2024-03-25T18:16:40.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.238+0000 7f9d27fff640 10 monclient: _finish_auth 0 2024-03-25T18:16:40.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.238+0000 7f9d27fff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:40.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.238+0000 7f9d27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:40.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.238+0000 7f9d376eb640 5 monclient: authenticate success, global_id 4607 2024-03-25T18:16:40.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.238+0000 7f9d376eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:40.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.244+0000 7f9d27fff640 1 -- 172.21.15.156:0/4264881786 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9d20001690 con 0x7f9d3013f710 2024-03-25T18:16:40.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.244+0000 7f9d27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:40.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.244+0000 7f9d27fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:40.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.244+0000 7f9d27fff640 10 monclient: dump: 2024-03-25T18:16:40.245 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:40.245 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:40.245 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:40.246 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:40.246 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:40.246 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:40.246 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:40.246 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:40.246 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:40.246 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:40.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.244+0000 7f9d376eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:40.247 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.245+0000 7f9d27fff640 1 -- 172.21.15.156:0/4264881786 <== mon.2 v1:172.21.15.156:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d20001920 con 0x7f9d3013f710 2024-03-25T18:16:40.247 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.245+0000 7f9d27fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:40.247 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.245+0000 7f9d27fff640 1 -- 172.21.15.156:0/4264881786 <== mon.2 v1:172.21.15.156:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9d20001c30 con 0x7f9d3013f710 2024-03-25T18:16:40.247 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.245+0000 7f9d27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:40.247 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.245+0000 7f9d27fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:40.247 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.245+0000 7f9d27fff640 10 monclient: dump: 2024-03-25T18:16:40.247 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:40.247 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:40.247 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:40.247 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:40.248 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:40.248 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:40.248 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:40.248 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:40.248 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:40.248 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:40.248 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.245+0000 7f9d376eb640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:40.248 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.245+0000 7f9d376eb640 4 set_mon_vals no callback set 2024-03-25T18:16:40.250 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.246+0000 7f9d376eb640 10 monclient: shutdown 2024-03-25T18:16:40.250 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.246+0000 7f9d376eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:40.250 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.246+0000 7f9d376eb640 1 -- 172.21.15.156:0/4264881786 >> v1:172.21.15.156:6790/0 conn(0x7f9d3013f710 legacy=0x7f9d301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:40.250 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.246+0000 7f9d376eb640 1 -- 172.21.15.156:0/4264881786 shutdown_connections 2024-03-25T18:16:40.250 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.246+0000 7f9d376eb640 1 -- 172.21.15.156:0/4264881786 >> 172.21.15.156:0/4264881786 conn(0x7f9d300b7ea0 msgr2=0x7f9d300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:40.250 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.246+0000 7f9d376eb640 1 -- 172.21.15.156:0/4264881786 shutdown_connections 2024-03-25T18:16:40.250 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.246+0000 7f9d376eb640 1 -- 172.21.15.156:0/4264881786 wait complete. 2024-03-25T18:16:40.251 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.247+0000 7f9d376eb640 10 monclient: build_initial_monmap 2024-03-25T18:16:40.251 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.247+0000 7f9d376eb640 10 monclient: monmap: 2024-03-25T18:16:40.251 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:16:40.251 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:40.251 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:16:40.251 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:16:40.251 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:40.251 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:16:40.251 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:16:40.251 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:16:40.251 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:16:40.251 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:40.251 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.247+0000 7f9d376eb640 1 librados: starting msgr at 2024-03-25T18:16:40.251 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.247+0000 7f9d376eb640 1 librados: starting objecter 2024-03-25T18:16:40.251 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.248+0000 7f9d376eb640 1 Processor -- start 2024-03-25T18:16:40.252 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.248+0000 7f9d376eb640 1 -- start start 2024-03-25T18:16:40.252 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.248+0000 7f9d376eb640 1 librados: setting wanted keys 2024-03-25T18:16:40.252 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.248+0000 7f9d376eb640 1 librados: calling monclient init 2024-03-25T18:16:40.252 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.248+0000 7f9d376eb640 10 monclient: init 2024-03-25T18:16:40.252 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.248+0000 7f9d376eb640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:40.252 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.248+0000 7f9d376eb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:40.252 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.248+0000 7f9d376eb640 10 monclient(hunting): picked mon.noname-b con 0x7f9d300a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:16:40.252 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.248+0000 7f9d376eb640 10 monclient(hunting): picked mon.noname-a con 0x7f9d3013f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:16:40.252 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.248+0000 7f9d376eb640 10 monclient(hunting): picked mon.noname-c con 0x7f9d301430c0 addr v1:172.21.15.156:6790/0 2024-03-25T18:16:40.252 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.248+0000 7f9d376eb640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d3006ffa0 con 0x7f9d3013f710 2024-03-25T18:16:40.252 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.248+0000 7f9d376eb640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d30070210 con 0x7f9d300a98f0 2024-03-25T18:16:40.252 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.248+0000 7f9d376eb640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d301399d0 con 0x7f9d301430c0 2024-03-25T18:16:40.252 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.248+0000 7f9d376eb640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:40.252 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.248+0000 7f9d376eb640 10 monclient(hunting): authenticate will time out at 7481.559719s 2024-03-25T18:16:40.259 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.258+0000 7f9d34c86640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f9d3013f710 0x7f9d30070630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:41798/0 (socket says 172.21.15.156:41798) 2024-03-25T18:16:40.259 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.258+0000 7f9d34c86640 1 -- 172.21.15.156:0/3270720912 learned_addr learned my addr 172.21.15.156:0/3270720912 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:16:40.259 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.258+0000 7f9d25ffb640 10 client.?.objecter ms_handle_connect 0x7f9d3013f710 2024-03-25T18:16:40.259 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.258+0000 7f9d25ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:40.265 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.264+0000 7f9d25ffb640 1 -- 172.21.15.156:0/3270720912 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 60855083 0 0) 0x7f9d3006ffa0 con 0x7f9d3013f710 2024-03-25T18:16:40.265 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.264+0000 7f9d25ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:40.265 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.264+0000 7f9d25ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:40.265 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.264+0000 7f9d25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:40.266 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.264+0000 7f9d25ffb640 10 monclient(hunting): my global_id is 4599 2024-03-25T18:16:40.266 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.264+0000 7f9d25ffb640 1 -- 172.21.15.156:0/3270720912 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d14004eb0 con 0x7f9d3013f710 2024-03-25T18:16:40.274 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.273+0000 7f9d25ffb640 10 client.?.objecter ms_handle_connect 0x7f9d300a98f0 2024-03-25T18:16:40.274 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.273+0000 7f9d25ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:40.279 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.278+0000 7f9d25ffb640 10 client.?.objecter ms_handle_connect 0x7f9d301430c0 2024-03-25T18:16:40.279 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.278+0000 7f9d25ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:40.281 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.280+0000 7f9d25ffb640 1 -- 172.21.15.156:0/3270720912 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2061964705 0 0) 0x7f9d14004eb0 con 0x7f9d3013f710 2024-03-25T18:16:40.281 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.280+0000 7f9d25ffb640 1 -- 172.21.15.156:0/3270720912 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d3006ffa0 con 0x7f9d3013f710 2024-03-25T18:16:40.281 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.280+0000 7f9d25ffb640 1 -- 172.21.15.156:0/3270720912 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9d20001650 con 0x7f9d3013f710 2024-03-25T18:16:40.281 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.280+0000 7f9d25ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:40.281 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.280+0000 7f9d25ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:40.281 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.280+0000 7f9d25ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:40.281 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:40.281 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:40.281 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:40.281 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:40.281 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:40.282 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:40.282 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:40.282 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:40.282 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:40.282 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:40.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.280+0000 7f9d25ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:40.289 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.288+0000 7f9d25ffb640 1 -- 172.21.15.156:0/3270720912 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2444749223 0 0) 0x7f9d30070210 con 0x7f9d300a98f0 2024-03-25T18:16:40.289 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.288+0000 7f9d25ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:40.289 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.288+0000 7f9d25ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:40.289 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.288+0000 7f9d25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:40.289 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.288+0000 7f9d25ffb640 10 monclient(hunting): my global_id is 4606 2024-03-25T18:16:40.290 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.288+0000 7f9d25ffb640 1 -- 172.21.15.156:0/3270720912 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d14004eb0 con 0x7f9d300a98f0 2024-03-25T18:16:40.290 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.288+0000 7f9d25ffb640 1 -- 172.21.15.156:0/3270720912 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1893264027 0 0) 0x7f9d301399d0 con 0x7f9d301430c0 2024-03-25T18:16:40.290 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.288+0000 7f9d25ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:40.290 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.288+0000 7f9d25ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:40.290 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.288+0000 7f9d25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:40.290 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.288+0000 7f9d25ffb640 10 monclient(hunting): my global_id is 4610 2024-03-25T18:16:40.290 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.288+0000 7f9d25ffb640 1 -- 172.21.15.156:0/3270720912 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d30070210 con 0x7f9d301430c0 2024-03-25T18:16:40.299 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.297+0000 7f9d25ffb640 1 -- 172.21.15.156:0/3270720912 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3274169971 0 0) 0x7f9d3006ffa0 con 0x7f9d3013f710 2024-03-25T18:16:40.299 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.297+0000 7f9d25ffb640 1 -- 172.21.15.156:0/3270720912 >> v1:172.21.15.156:6790/0 conn(0x7f9d301430c0 legacy=0x7f9d301e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:40.299 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.297+0000 7f9d25ffb640 1 -- 172.21.15.156:0/3270720912 >> v1:172.21.15.194:6789/0 conn(0x7f9d300a98f0 legacy=0x7f9d3006ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:40.299 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.297+0000 7f9d25ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:40.299 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.297+0000 7f9d25ffb640 1 monclient: found mon.a 2024-03-25T18:16:40.299 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.297+0000 7f9d25ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:40.299 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.297+0000 7f9d25ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:16:40.299 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.297+0000 7f9d25ffb640 1 -- 172.21.15.156:0/3270720912 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3014e610 con 0x7f9d3013f710 2024-03-25T18:16:40.299 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.297+0000 7f9d25ffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:40.299 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.297+0000 7f9d25ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:40.300 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.297+0000 7f9d25ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:40.300 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.297+0000 7f9d376eb640 5 monclient: authenticate success, global_id 4599 2024-03-25T18:16:40.300 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.297+0000 7f9d376eb640 10 monclient: _renew_subs 2024-03-25T18:16:40.300 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.297+0000 7f9d376eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:16:40.300 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.297+0000 7f9d376eb640 1 -- 172.21.15.156:0/3270720912 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d3014d1d0 con 0x7f9d3013f710 2024-03-25T18:16:40.300 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.298+0000 7f9d376eb640 10 client.4599.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:40.300 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.298+0000 7f9d376eb640 10 monclient: _renew_subs 2024-03-25T18:16:40.300 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.298+0000 7f9d376eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:16:40.300 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.298+0000 7f9d376eb640 1 -- 172.21.15.156:0/3270720912 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d3006e2f0 con 0x7f9d3013f710 2024-03-25T18:16:40.300 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.298+0000 7f9d376eb640 1 librados: init done 2024-03-25T18:16:40.301 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.299+0000 7f9d376eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:40.301 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.299+0000 7f9d376eb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:40.301 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.299+0000 7f9d376eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:16:40.301 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.299+0000 7f9d376eb640 1 -- 172.21.15.156:0/3270720912 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d3006e5a0 con 0x7f9d3013f710 2024-03-25T18:16:40.591 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.590+0000 7f9d25ffb640 1 -- 172.21.15.156:0/3270720912 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d20004bf0 con 0x7f9d3013f710 2024-03-25T18:16:40.591 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.590+0000 7f9d25ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:40.591 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.590+0000 7f9d25ffb640 1 -- 172.21.15.156:0/3270720912 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9d20004f00 con 0x7f9d3013f710 2024-03-25T18:16:40.591 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.590+0000 7f9d25ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:40.591 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.590+0000 7f9d25ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:40.591 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.590+0000 7f9d25ffb640 10 monclient: dump: 2024-03-25T18:16:40.591 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:40.591 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:40.591 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:40.591 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:40.591 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:40.592 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:40.592 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:40.592 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:40.592 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:40.592 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:40.592 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.590+0000 7f9d277fe640 4 set_mon_vals no callback set 2024-03-25T18:16:40.592 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.590+0000 7f9d25ffb640 1 -- 172.21.15.156:0/3270720912 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f9d2001aed0 con 0x7f9d3013f710 2024-03-25T18:16:40.593 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.591+0000 7f9d35487640 10 monclient: get_auth_request con 0x7f9d1406de60 auth_method 0 2024-03-25T18:16:40.593 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.591+0000 7f9d25ffb640 10 client.4599.objecter ms_handle_connect 0x7f9d1406de60 2024-03-25T18:16:40.597 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.596+0000 7f9d25ffb640 1 -- 172.21.15.156:0/3270720912 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f9d20054970 con 0x7f9d3013f710 2024-03-25T18:16:40.597 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.596+0000 7f9d25ffb640 10 client.4599.objecter ms_dispatch 0x7f9d30000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:40.597 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.596+0000 7f9d25ffb640 3 client.4599.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:40.597 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.596+0000 7f9d25ffb640 3 client.4599.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:40.597 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.596+0000 7f9d25ffb640 20 client.4599.objecter dump_active .. 0 homeless 2024-03-25T18:16:40.597 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.596+0000 7f9d25ffb640 1 -- 172.21.15.156:0/3270720912 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9d20050480 con 0x7f9d3013f710 2024-03-25T18:16:40.597 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.596+0000 7f9d25ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:40.598 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.596+0000 7f9d25ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:40.840 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.839+0000 7f9d376eb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:40.840 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.839+0000 7f9d376eb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:40.840 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.839+0000 7f9d376eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:16:40.840 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.839+0000 7f9d376eb640 1 -- 172.21.15.156:0/3270720912 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d300627b0 con 0x7f9d3013f710 2024-03-25T18:16:40.921 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.919+0000 7f9d25ffb640 1 -- 172.21.15.156:0/3270720912 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f9d200523b0 con 0x7f9d3013f710 2024-03-25T18:16:40.921 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.919+0000 7f9d25ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:40.921 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.919+0000 7f9d25ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:40.921 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:16:40.921 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:40.923 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.921+0000 7f9d376eb640 10 librados: watch_flush enter 2024-03-25T18:16:40.923 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.921+0000 7f9d376eb640 10 librados: watch_flush exit 2024-03-25T18:16:40.924 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.922+0000 7f9d376eb640 20 client.4599.objecter shutdown clearing up homeless session... 2024-03-25T18:16:40.924 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.922+0000 7f9d376eb640 10 client.4599.objecter successfully canceled tick 2024-03-25T18:16:40.924 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.922+0000 7f9d376eb640 1 -- 172.21.15.156:0/3270720912 >> v1:172.21.15.194:6800/2946427713 conn(0x7f9d1406de60 legacy=0x7f9d14070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:40.924 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.922+0000 7f9d376eb640 10 monclient: shutdown 2024-03-25T18:16:40.924 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.922+0000 7f9d376eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:40.924 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.922+0000 7f9d376eb640 1 -- 172.21.15.156:0/3270720912 >> v1:172.21.15.156:6789/0 conn(0x7f9d3013f710 legacy=0x7f9d30070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:40.924 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.922+0000 7f9d376eb640 1 -- 172.21.15.156:0/3270720912 shutdown_connections 2024-03-25T18:16:40.924 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.922+0000 7f9d376eb640 1 -- 172.21.15.156:0/3270720912 >> 172.21.15.156:0/3270720912 conn(0x7f9d300b7ea0 msgr2=0x7f9d30074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:40.924 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.922+0000 7f9d376eb640 1 -- 172.21.15.156:0/3270720912 shutdown_connections 2024-03-25T18:16:40.924 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.922+0000 7f9d376eb640 1 -- 172.21.15.156:0/3270720912 wait complete. 2024-03-25T18:16:40.924 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:40.922+0000 7f9d376eb640 1 librados: shutdown 2024-03-25T18:16:40.934 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:47.740 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:47.855 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.854+0000 7f44f6ff4640 10 monclient: get_monmap_and_config 2024-03-25T18:16:47.856 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.854+0000 7f44f6ff4640 10 monclient: build_initial_monmap 2024-03-25T18:16:47.856 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.854+0000 7f44f6ff4640 10 monclient: monmap: 2024-03-25T18:16:47.856 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:16:47.856 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:47.856 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:16:47.855775+0000 2024-03-25T18:16:47.856 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:16:47.855775+0000 2024-03-25T18:16:47.856 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:47.856 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:47.856 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:16:47.856 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:16:47.856 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:16:47.856 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:47.857 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.855+0000 7f44f6ff4640 1 Processor -- start 2024-03-25T18:16:47.857 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.855+0000 7f44f6ff4640 1 -- start start 2024-03-25T18:16:47.857 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.855+0000 7f44f6ff4640 10 monclient: init 2024-03-25T18:16:47.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.856+0000 7f44f6ff4640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:47.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.856+0000 7f44f6ff4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:47.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.856+0000 7f44f6ff4640 10 monclient(hunting): picked mon.noname-c con 0x7f44f00a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:16:47.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.856+0000 7f44f6ff4640 10 monclient(hunting): picked mon.noname-b con 0x7f44f013f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:16:47.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.856+0000 7f44f6ff4640 10 monclient(hunting): picked mon.noname-a con 0x7f44f01430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:16:47.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.856+0000 7f44f6ff4640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f0142520 con 0x7f44f01430c0 2024-03-25T18:16:47.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.856+0000 7f44f6ff4640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f00b28a0 con 0x7f44f00a98f0 2024-03-25T18:16:47.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.856+0000 7f44f6ff4640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f00b2a30 con 0x7f44f013f710 2024-03-25T18:16:47.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.856+0000 7f44f6ff4640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:47.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.856+0000 7f44f6ff4640 10 monclient(hunting): authenticate will time out at 7489.167559s 2024-03-25T18:16:47.868 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.867+0000 7f44f4d90640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f44f00a98f0 0x7f44f00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:44398/0 (socket says 172.21.15.156:44398) 2024-03-25T18:16:47.868 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.867+0000 7f44f4d90640 1 -- 172.21.15.156:0/3997418023 learned_addr learned my addr 172.21.15.156:0/3997418023 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:16:47.882 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.881+0000 7f44ef7fe640 1 -- 172.21.15.156:0/3997418023 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 905757253 0 0) 0x7f44f00b28a0 con 0x7f44f00a98f0 2024-03-25T18:16:47.882 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.881+0000 7f44ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:47.882 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.881+0000 7f44ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:47.882 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.881+0000 7f44ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:47.882 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.881+0000 7f44ef7fe640 10 monclient(hunting): my global_id is 4609 2024-03-25T18:16:47.882 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.881+0000 7f44ef7fe640 1 -- 172.21.15.156:0/3997418023 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44e0003520 con 0x7f44f00a98f0 2024-03-25T18:16:47.896 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.895+0000 7f44ef7fe640 1 -- 172.21.15.156:0/3997418023 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3671798732 0 0) 0x7f44f00b2a30 con 0x7f44f013f710 2024-03-25T18:16:47.896 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.895+0000 7f44ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:47.896 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.895+0000 7f44ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:47.897 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.895+0000 7f44ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:47.897 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.895+0000 7f44ef7fe640 10 monclient(hunting): my global_id is 4613 2024-03-25T18:16:47.897 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.895+0000 7f44ef7fe640 1 -- 172.21.15.156:0/3997418023 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44f00b28a0 con 0x7f44f013f710 2024-03-25T18:16:47.905 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.904+0000 7f44ef7fe640 1 -- 172.21.15.156:0/3997418023 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2733178989 0 0) 0x7f44e0003520 con 0x7f44f00a98f0 2024-03-25T18:16:47.905 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.904+0000 7f44ef7fe640 1 -- 172.21.15.156:0/3997418023 >> v1:172.21.15.156:6790/0 conn(0x7f44f013f710 legacy=0x7f44f01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:47.905 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.904+0000 7f44ef7fe640 1 -- 172.21.15.156:0/3997418023 >> v1:172.21.15.156:6789/0 conn(0x7f44f01430c0 legacy=0x7f44f014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:47.905 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.904+0000 7f44ef7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:47.905 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.904+0000 7f44ef7fe640 1 monclient: found mon.noname-c 2024-03-25T18:16:47.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.904+0000 7f44ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:47.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.904+0000 7f44ef7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:16:47.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.904+0000 7f44ef7fe640 1 -- 172.21.15.156:0/3997418023 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f00b44c0 con 0x7f44f00a98f0 2024-03-25T18:16:47.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.904+0000 7f44ef7fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:47.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.904+0000 7f44ef7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:47.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.904+0000 7f44ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:47.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.904+0000 7f44ef7fe640 1 -- 172.21.15.156:0/3997418023 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f44e4001690 con 0x7f44f00a98f0 2024-03-25T18:16:47.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.904+0000 7f44ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:47.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.904+0000 7f44ef7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:47.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.904+0000 7f44ef7fe640 10 monclient: dump: 2024-03-25T18:16:47.906 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:47.906 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:47.906 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:47.906 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:47.906 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:47.907 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:47.907 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:47.907 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:47.907 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:47.907 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:47.907 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.904+0000 7f44f6ff4640 5 monclient: authenticate success, global_id 4609 2024-03-25T18:16:47.907 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.904+0000 7f44f6ff4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:47.909 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.908+0000 7f44ef7fe640 1 -- 172.21.15.156:0/3997418023 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44e40017f0 con 0x7f44f00a98f0 2024-03-25T18:16:47.909 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.908+0000 7f44ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:47.909 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.908+0000 7f44ef7fe640 1 -- 172.21.15.156:0/3997418023 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f44e4001c00 con 0x7f44f00a98f0 2024-03-25T18:16:47.909 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.908+0000 7f44ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:47.910 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.908+0000 7f44ef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:47.910 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.908+0000 7f44ef7fe640 10 monclient: dump: 2024-03-25T18:16:47.910 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:47.910 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:47.910 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:47.910 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:47.910 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:47.910 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:47.910 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:47.910 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:47.910 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:47.910 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:47.910 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.908+0000 7f44f6ff4640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:47.910 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.908+0000 7f44f6ff4640 4 set_mon_vals no callback set 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.909+0000 7f44f6ff4640 10 monclient: shutdown 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.909+0000 7f44f6ff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.909+0000 7f44f6ff4640 1 -- 172.21.15.156:0/3997418023 >> v1:172.21.15.194:6789/0 conn(0x7f44f00a98f0 legacy=0x7f44f00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.909+0000 7f44f6ff4640 1 -- 172.21.15.156:0/3997418023 shutdown_connections 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.909+0000 7f44f6ff4640 1 -- 172.21.15.156:0/3997418023 >> 172.21.15.156:0/3997418023 conn(0x7f44f00b7ea0 msgr2=0x7f44f00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.909+0000 7f44f6ff4640 1 -- 172.21.15.156:0/3997418023 shutdown_connections 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.909+0000 7f44f6ff4640 1 -- 172.21.15.156:0/3997418023 wait complete. 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.910+0000 7f44f6ff4640 10 monclient: build_initial_monmap 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.910+0000 7f44f6ff4640 10 monclient: monmap: 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:47.913 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:16:47.913 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:16:47.913 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:16:47.913 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:16:47.913 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:47.913 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.910+0000 7f44f6ff4640 1 librados: starting msgr at 2024-03-25T18:16:47.913 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.910+0000 7f44f6ff4640 1 librados: starting objecter 2024-03-25T18:16:47.913 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.910+0000 7f44f6ff4640 1 Processor -- start 2024-03-25T18:16:47.913 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.911+0000 7f44f6ff4640 1 -- start start 2024-03-25T18:16:47.913 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.911+0000 7f44f6ff4640 1 librados: setting wanted keys 2024-03-25T18:16:47.913 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.911+0000 7f44f6ff4640 1 librados: calling monclient init 2024-03-25T18:16:47.913 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.911+0000 7f44f6ff4640 10 monclient: init 2024-03-25T18:16:47.914 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.911+0000 7f44f6ff4640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:47.914 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.911+0000 7f44f6ff4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:47.914 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.911+0000 7f44f6ff4640 10 monclient(hunting): picked mon.noname-c con 0x7f44f013f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:16:47.914 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.911+0000 7f44f6ff4640 10 monclient(hunting): picked mon.noname-b con 0x7f44f01430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:16:47.914 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.911+0000 7f44f6ff4640 10 monclient(hunting): picked mon.noname-a con 0x7f44f00ad4c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:16:47.915 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.911+0000 7f44f6ff4640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f00b0720 con 0x7f44f00ad4c0 2024-03-25T18:16:47.915 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.911+0000 7f44f6ff4640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f00b1b60 con 0x7f44f01430c0 2024-03-25T18:16:47.915 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.912+0000 7f44f6ff4640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f00b1e40 con 0x7f44f013f710 2024-03-25T18:16:47.915 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.912+0000 7f44f6ff4640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:47.915 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.912+0000 7f44f6ff4640 10 monclient(hunting): authenticate will time out at 7489.222883s 2024-03-25T18:16:47.919 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.918+0000 7f44effff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f44f01430c0 0x7f44f01df780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:44442/0 (socket says 172.21.15.156:44442) 2024-03-25T18:16:47.919 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.918+0000 7f44effff640 1 -- 172.21.15.156:0/4140156261 learned_addr learned my addr 172.21.15.156:0/4140156261 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:16:47.922 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.921+0000 7f44ed7fa640 10 client.?.objecter ms_handle_connect 0x7f44f01430c0 2024-03-25T18:16:47.922 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.921+0000 7f44ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:47.928 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.927+0000 7f44ed7fa640 10 client.?.objecter ms_handle_connect 0x7f44f00ad4c0 2024-03-25T18:16:47.928 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.927+0000 7f44ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:47.931 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.930+0000 7f44ed7fa640 10 client.?.objecter ms_handle_connect 0x7f44f013f710 2024-03-25T18:16:47.931 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.930+0000 7f44ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:47.936 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.935+0000 7f44ed7fa640 1 -- 172.21.15.156:0/4140156261 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230172605 0 0) 0x7f44f00b0720 con 0x7f44f00ad4c0 2024-03-25T18:16:47.936 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.935+0000 7f44ed7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:47.936 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.935+0000 7f44ed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:47.936 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.935+0000 7f44ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:47.936 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.935+0000 7f44ed7fa640 10 monclient(hunting): my global_id is 4605 2024-03-25T18:16:47.936 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.935+0000 7f44ed7fa640 1 -- 172.21.15.156:0/4140156261 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44d4004eb0 con 0x7f44f00ad4c0 2024-03-25T18:16:47.942 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.941+0000 7f44ed7fa640 1 -- 172.21.15.156:0/4140156261 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2186409163 0 0) 0x7f44f00b1b60 con 0x7f44f01430c0 2024-03-25T18:16:47.942 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.941+0000 7f44ed7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:47.942 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.941+0000 7f44ed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:47.942 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.941+0000 7f44ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:47.942 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.941+0000 7f44ed7fa640 10 monclient(hunting): my global_id is 4612 2024-03-25T18:16:47.942 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.941+0000 7f44ed7fa640 1 -- 172.21.15.156:0/4140156261 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44f00b0720 con 0x7f44f01430c0 2024-03-25T18:16:47.949 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.948+0000 7f44ed7fa640 1 -- 172.21.15.156:0/4140156261 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3236615004 0 0) 0x7f44d4004eb0 con 0x7f44f00ad4c0 2024-03-25T18:16:47.949 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.948+0000 7f44ed7fa640 1 -- 172.21.15.156:0/4140156261 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44f00b1b60 con 0x7f44f00ad4c0 2024-03-25T18:16:47.949 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.948+0000 7f44ed7fa640 1 -- 172.21.15.156:0/4140156261 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f44e8001b90 con 0x7f44f00ad4c0 2024-03-25T18:16:47.949 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.948+0000 7f44ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:47.949 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.948+0000 7f44ed7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:47.949 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.948+0000 7f44ed7fa640 10 monclient(hunting): dump: 2024-03-25T18:16:47.949 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:47.949 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:47.949 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:47.949 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:47.949 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:47.950 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:47.950 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:47.950 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:47.950 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:47.950 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:47.950 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.948+0000 7f44ed7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:47.953 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.952+0000 7f44ed7fa640 1 -- 172.21.15.156:0/4140156261 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205910713 0 0) 0x7f44f00b1e40 con 0x7f44f013f710 2024-03-25T18:16:47.953 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.952+0000 7f44ed7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:47.954 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.952+0000 7f44ed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:47.954 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.952+0000 7f44ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:47.954 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.952+0000 7f44ed7fa640 10 monclient(hunting): my global_id is 4616 2024-03-25T18:16:47.954 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.952+0000 7f44ed7fa640 1 -- 172.21.15.156:0/4140156261 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44d4004eb0 con 0x7f44f013f710 2024-03-25T18:16:47.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.959+0000 7f44ed7fa640 1 -- 172.21.15.156:0/4140156261 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 706423219 0 0) 0x7f44f00b0720 con 0x7f44f01430c0 2024-03-25T18:16:47.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.959+0000 7f44ed7fa640 1 -- 172.21.15.156:0/4140156261 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44f00b1e40 con 0x7f44f01430c0 2024-03-25T18:16:47.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.959+0000 7f44ed7fa640 1 -- 172.21.15.156:0/4140156261 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f44dc0019f0 con 0x7f44f01430c0 2024-03-25T18:16:47.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.959+0000 7f44ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:47.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.959+0000 7f44ed7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:47.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.959+0000 7f44ed7fa640 10 monclient(hunting): dump: 2024-03-25T18:16:47.961 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:47.961 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:47.961 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:47.961 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:47.961 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:47.961 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:47.961 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:47.961 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:47.961 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:47.962 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:47.966 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.964+0000 7f44ed7fa640 1 -- 172.21.15.156:0/4140156261 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2268819787 0 0) 0x7f44f00b1b60 con 0x7f44f00ad4c0 2024-03-25T18:16:47.966 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.964+0000 7f44ed7fa640 1 -- 172.21.15.156:0/4140156261 >> v1:172.21.15.156:6790/0 conn(0x7f44f013f710 legacy=0x7f44f00acdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:47.966 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.964+0000 7f44ed7fa640 1 -- 172.21.15.156:0/4140156261 >> v1:172.21.15.194:6789/0 conn(0x7f44f01430c0 legacy=0x7f44f01df780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:47.966 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.964+0000 7f44ed7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:16:47.966 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.964+0000 7f44ed7fa640 1 monclient: found mon.a 2024-03-25T18:16:47.966 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.964+0000 7f44ed7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:47.966 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.964+0000 7f44ed7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:16:47.966 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.964+0000 7f44ed7fa640 1 -- 172.21.15.156:0/4140156261 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f00b0bb0 con 0x7f44f00ad4c0 2024-03-25T18:16:47.966 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.964+0000 7f44ed7fa640 10 monclient: _finish_auth 0 2024-03-25T18:16:47.966 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.964+0000 7f44ed7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:16:47.966 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.964+0000 7f44ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:47.966 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.964+0000 7f44f6ff4640 5 monclient: authenticate success, global_id 4605 2024-03-25T18:16:47.966 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.964+0000 7f44f6ff4640 10 monclient: _renew_subs 2024-03-25T18:16:47.967 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.964+0000 7f44f6ff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:16:47.967 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.964+0000 7f44f6ff4640 1 -- 172.21.15.156:0/4140156261 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f00b0130 con 0x7f44f00ad4c0 2024-03-25T18:16:47.967 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.964+0000 7f44f6ff4640 10 client.4605.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:47.967 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.964+0000 7f44f6ff4640 10 monclient: _renew_subs 2024-03-25T18:16:47.967 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.964+0000 7f44f6ff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:16:47.967 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.964+0000 7f44f6ff4640 1 -- 172.21.15.156:0/4140156261 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f00af7e0 con 0x7f44f00ad4c0 2024-03-25T18:16:47.967 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.964+0000 7f44f6ff4640 1 librados: init done 2024-03-25T18:16:47.967 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.965+0000 7f44f6ff4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:47.968 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.965+0000 7f44f6ff4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:47.968 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.965+0000 7f44f6ff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:16:47.968 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:47.965+0000 7f44f6ff4640 1 -- 172.21.15.156:0/4140156261 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f00b3ef0 con 0x7f44f00ad4c0 2024-03-25T18:16:48.270 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.269+0000 7f44ed7fa640 1 -- 172.21.15.156:0/4140156261 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44e8002080 con 0x7f44f00ad4c0 2024-03-25T18:16:48.271 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.269+0000 7f44ed7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:48.271 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.269+0000 7f44ed7fa640 1 -- 172.21.15.156:0/4140156261 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f44e8003b10 con 0x7f44f00ad4c0 2024-03-25T18:16:48.271 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.269+0000 7f44ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:48.271 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.269+0000 7f44ed7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:48.271 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.269+0000 7f44ed7fa640 10 monclient: dump: 2024-03-25T18:16:48.271 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:48.271 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:48.271 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:48.271 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:48.271 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:48.271 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:48.271 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:48.271 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:48.272 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:48.272 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:48.272 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.269+0000 7f44eeffd640 4 set_mon_vals no callback set 2024-03-25T18:16:48.272 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.270+0000 7f44ed7fa640 1 -- 172.21.15.156:0/4140156261 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f44e8003d30 con 0x7f44f00ad4c0 2024-03-25T18:16:48.272 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.271+0000 7f44f4d90640 10 monclient: get_auth_request con 0x7f44d406dfb0 auth_method 0 2024-03-25T18:16:48.272 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.271+0000 7f44ed7fa640 10 client.4605.objecter ms_handle_connect 0x7f44d406dfb0 2024-03-25T18:16:48.279 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.278+0000 7f44ed7fa640 1 -- 172.21.15.156:0/4140156261 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f44e80538c0 con 0x7f44f00ad4c0 2024-03-25T18:16:48.279 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.278+0000 7f44ed7fa640 10 client.4605.objecter ms_dispatch 0x7f44f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:48.280 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.278+0000 7f44ed7fa640 3 client.4605.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:48.280 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.278+0000 7f44ed7fa640 3 client.4605.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:48.280 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.278+0000 7f44ed7fa640 20 client.4605.objecter dump_active .. 0 homeless 2024-03-25T18:16:48.280 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.278+0000 7f44ed7fa640 1 -- 172.21.15.156:0/4140156261 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f44e804f200 con 0x7f44f00ad4c0 2024-03-25T18:16:48.280 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.278+0000 7f44ed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:48.280 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.278+0000 7f44ed7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:48.498 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.497+0000 7f44f6ff4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:48.498 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.497+0000 7f44f6ff4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:48.498 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.497+0000 7f44f6ff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:16:48.498 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.497+0000 7f44f6ff4640 1 -- 172.21.15.156:0/4140156261 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44f00627b0 con 0x7f44f00ad4c0 2024-03-25T18:16:48.545 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.543+0000 7f44ed7fa640 1 -- 172.21.15.156:0/4140156261 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f44e8051300 con 0x7f44f00ad4c0 2024-03-25T18:16:48.545 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.543+0000 7f44ed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:48.545 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.543+0000 7f44ed7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:48.545 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:16:48.545 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:48.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.546+0000 7f44f6ff4640 10 librados: watch_flush enter 2024-03-25T18:16:48.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.546+0000 7f44f6ff4640 10 librados: watch_flush exit 2024-03-25T18:16:48.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.546+0000 7f44f6ff4640 20 client.4605.objecter shutdown clearing up homeless session... 2024-03-25T18:16:48.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.546+0000 7f44f6ff4640 10 client.4605.objecter successfully canceled tick 2024-03-25T18:16:48.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.546+0000 7f44f6ff4640 1 -- 172.21.15.156:0/4140156261 >> v1:172.21.15.194:6800/2946427713 conn(0x7f44d406dfb0 legacy=0x7f44d4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:48.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.546+0000 7f44f6ff4640 10 monclient: shutdown 2024-03-25T18:16:48.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.546+0000 7f44f6ff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:48.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.546+0000 7f44f6ff4640 1 -- 172.21.15.156:0/4140156261 >> v1:172.21.15.156:6789/0 conn(0x7f44f00ad4c0 legacy=0x7f44f01e2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:48.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.546+0000 7f44f6ff4640 1 -- 172.21.15.156:0/4140156261 shutdown_connections 2024-03-25T18:16:48.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.546+0000 7f44f6ff4640 1 -- 172.21.15.156:0/4140156261 >> 172.21.15.156:0/4140156261 conn(0x7f44f00b7ea0 msgr2=0x7f44f013bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:48.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.546+0000 7f44f6ff4640 1 -- 172.21.15.156:0/4140156261 shutdown_connections 2024-03-25T18:16:48.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.546+0000 7f44f6ff4640 1 -- 172.21.15.156:0/4140156261 wait complete. 2024-03-25T18:16:48.549 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:48.546+0000 7f44f6ff4640 1 librados: shutdown 2024-03-25T18:16:48.558 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:55.364 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:55.476 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.475+0000 7f8398e0f640 10 monclient: get_monmap_and_config 2024-03-25T18:16:55.476 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.475+0000 7f8398e0f640 10 monclient: build_initial_monmap 2024-03-25T18:16:55.476 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.475+0000 7f8398e0f640 10 monclient: monmap: 2024-03-25T18:16:55.476 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:16:55.476 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:55.476 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:16:55.476312+0000 2024-03-25T18:16:55.476 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:16:55.476312+0000 2024-03-25T18:16:55.477 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:55.477 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:55.477 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:16:55.477 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:16:55.477 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:16:55.477 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:55.477 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.476+0000 7f8398e0f640 1 Processor -- start 2024-03-25T18:16:55.477 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.476+0000 7f8398e0f640 1 -- start start 2024-03-25T18:16:55.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.476+0000 7f8398e0f640 10 monclient: init 2024-03-25T18:16:55.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.477+0000 7f8398e0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:55.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.477+0000 7f8398e0f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:55.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.477+0000 7f8398e0f640 10 monclient(hunting): picked mon.noname-a con 0x7f83940a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:16:55.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.477+0000 7f8398e0f640 10 monclient(hunting): picked mon.noname-b con 0x7f839413f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:16:55.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.477+0000 7f8398e0f640 10 monclient(hunting): picked mon.noname-c con 0x7f83941430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:16:55.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.477+0000 7f8398e0f640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8394142520 con 0x7f83940a98f0 2024-03-25T18:16:55.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.477+0000 7f8398e0f640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83940b28a0 con 0x7f83941430c0 2024-03-25T18:16:55.479 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.477+0000 7f8398e0f640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83940b2a30 con 0x7f839413f710 2024-03-25T18:16:55.479 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.477+0000 7f8398e0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:55.479 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.477+0000 7f8398e0f640 10 monclient(hunting): authenticate will time out at 7496.787996s 2024-03-25T18:16:55.494 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.493+0000 7f839259c640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f83940a98f0 0x7f83940a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:57362/0 (socket says 172.21.15.156:57362) 2024-03-25T18:16:55.494 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.493+0000 7f839259c640 1 -- 172.21.15.156:0/2345328240 learned_addr learned my addr 172.21.15.156:0/2345328240 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:16:55.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.507+0000 7f8391d9b640 1 -- 172.21.15.156:0/2345328240 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287522445 0 0) 0x7f83940b28a0 con 0x7f83941430c0 2024-03-25T18:16:55.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.507+0000 7f8391d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:16:55.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.507+0000 7f8391d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:55.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.507+0000 7f8391d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:55.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.507+0000 7f8391d9b640 10 monclient(hunting): my global_id is 4615 2024-03-25T18:16:55.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.507+0000 7f8391d9b640 1 -- 172.21.15.156:0/2345328240 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8380003520 con 0x7f83941430c0 2024-03-25T18:16:55.516 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.515+0000 7f8391d9b640 1 -- 172.21.15.156:0/2345328240 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 482102121 0 0) 0x7f8380003520 con 0x7f83941430c0 2024-03-25T18:16:55.516 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.515+0000 7f8391d9b640 1 -- 172.21.15.156:0/2345328240 >> v1:172.21.15.156:6790/0 conn(0x7f839413f710 legacy=0x7f83941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:55.516 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.515+0000 7f8391d9b640 1 -- 172.21.15.156:0/2345328240 >> v1:172.21.15.156:6789/0 conn(0x7f83940a98f0 legacy=0x7f83940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:55.516 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.515+0000 7f8391d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:16:55.516 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.515+0000 7f8391d9b640 1 monclient: found mon.noname-c 2024-03-25T18:16:55.516 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.515+0000 7f8391d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:55.517 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.515+0000 7f8391d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:16:55.517 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.515+0000 7f8391d9b640 1 -- 172.21.15.156:0/2345328240 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83940b44c0 con 0x7f83941430c0 2024-03-25T18:16:55.517 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.515+0000 7f8391d9b640 10 monclient: _finish_auth 0 2024-03-25T18:16:55.517 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.515+0000 7f8391d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:16:55.517 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.515+0000 7f8391d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:55.517 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.515+0000 7f8391d9b640 1 -- 172.21.15.156:0/2345328240 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f838c001a10 con 0x7f83941430c0 2024-03-25T18:16:55.517 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.515+0000 7f8391d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:55.517 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.515+0000 7f8391d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:55.517 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.515+0000 7f8391d9b640 10 monclient: dump: 2024-03-25T18:16:55.517 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:55.517 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:55.517 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:55.517 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:55.517 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:55.517 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:55.518 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:55.518 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:55.518 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:55.518 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:55.518 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.515+0000 7f8398e0f640 5 monclient: authenticate success, global_id 4615 2024-03-25T18:16:55.518 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.515+0000 7f8398e0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:55.532 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.530+0000 7f8391d9b640 1 -- 172.21.15.156:0/2345328240 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f838c001b70 con 0x7f83941430c0 2024-03-25T18:16:55.532 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.530+0000 7f8391d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:55.532 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.530+0000 7f8391d9b640 1 -- 172.21.15.156:0/2345328240 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f838c001f80 con 0x7f83941430c0 2024-03-25T18:16:55.532 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.530+0000 7f8391d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:55.532 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.530+0000 7f8391d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:55.532 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.530+0000 7f8391d9b640 10 monclient: dump: 2024-03-25T18:16:55.532 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:55.532 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:55.532 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:55.532 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:55.532 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:55.532 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:55.532 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:55.533 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:55.533 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:55.533 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:55.533 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.531+0000 7f8398e0f640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:55.533 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.531+0000 7f8398e0f640 4 set_mon_vals no callback set 2024-03-25T18:16:55.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.531+0000 7f8398e0f640 10 monclient: shutdown 2024-03-25T18:16:55.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.531+0000 7f8398e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:55.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.531+0000 7f8398e0f640 1 -- 172.21.15.156:0/2345328240 >> v1:172.21.15.194:6789/0 conn(0x7f83941430c0 legacy=0x7f839414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:55.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.531+0000 7f8398e0f640 1 -- 172.21.15.156:0/2345328240 shutdown_connections 2024-03-25T18:16:55.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.531+0000 7f8398e0f640 1 -- 172.21.15.156:0/2345328240 >> 172.21.15.156:0/2345328240 conn(0x7f83940b7ea0 msgr2=0x7f83940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:55.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.531+0000 7f8398e0f640 1 -- 172.21.15.156:0/2345328240 shutdown_connections 2024-03-25T18:16:55.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.531+0000 7f8398e0f640 1 -- 172.21.15.156:0/2345328240 wait complete. 2024-03-25T18:16:55.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.532+0000 7f8398e0f640 10 monclient: build_initial_monmap 2024-03-25T18:16:55.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.532+0000 7f8398e0f640 10 monclient: monmap: 2024-03-25T18:16:55.535 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:16:55.536 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:55.536 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:16:55.536 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:16:55.536 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:55.536 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:16:55.536 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:16:55.536 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:16:55.536 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:16:55.536 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:55.536 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.532+0000 7f8398e0f640 1 librados: starting msgr at 2024-03-25T18:16:55.536 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.532+0000 7f8398e0f640 1 librados: starting objecter 2024-03-25T18:16:55.536 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.533+0000 7f8398e0f640 1 Processor -- start 2024-03-25T18:16:55.536 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.533+0000 7f8398e0f640 1 -- start start 2024-03-25T18:16:55.536 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.533+0000 7f8398e0f640 1 librados: setting wanted keys 2024-03-25T18:16:55.536 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.533+0000 7f8398e0f640 1 librados: calling monclient init 2024-03-25T18:16:55.537 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.533+0000 7f8398e0f640 10 monclient: init 2024-03-25T18:16:55.537 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.533+0000 7f8398e0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:55.537 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.533+0000 7f8398e0f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:55.537 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.533+0000 7f8398e0f640 10 monclient(hunting): picked mon.noname-c con 0x7f83940a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:16:55.537 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.533+0000 7f8398e0f640 10 monclient(hunting): picked mon.noname-b con 0x7f839413f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:16:55.537 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.533+0000 7f8398e0f640 10 monclient(hunting): picked mon.noname-a con 0x7f83941430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:16:55.537 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.533+0000 7f8398e0f640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f839413cc30 con 0x7f83941430c0 2024-03-25T18:16:55.537 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.533+0000 7f8398e0f640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83940b71e0 con 0x7f839413f710 2024-03-25T18:16:55.537 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.533+0000 7f8398e0f640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f839413afc0 con 0x7f83940a98f0 2024-03-25T18:16:55.537 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.533+0000 7f8398e0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:55.537 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.533+0000 7f8398e0f640 10 monclient(hunting): authenticate will time out at 7496.844472s 2024-03-25T18:16:55.544 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.543+0000 7f838bfff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f839413f710 0x7f8394139e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:40004/0 (socket says 172.21.15.156:40004) 2024-03-25T18:16:55.544 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.543+0000 7f838bfff640 1 -- 172.21.15.156:0/271982395 learned_addr learned my addr 172.21.15.156:0/271982395 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:16:55.551 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.550+0000 7f838affd640 10 client.?.objecter ms_handle_connect 0x7f839413f710 2024-03-25T18:16:55.551 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.550+0000 7f838affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:55.553 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.552+0000 7f838affd640 10 client.?.objecter ms_handle_connect 0x7f83940a98f0 2024-03-25T18:16:55.553 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.552+0000 7f838affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:55.560 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.559+0000 7f838affd640 10 client.?.objecter ms_handle_connect 0x7f83941430c0 2024-03-25T18:16:55.560 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.559+0000 7f838affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:55.562 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.561+0000 7f838affd640 1 -- 172.21.15.156:0/271982395 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778403750 0 0) 0x7f83940b71e0 con 0x7f839413f710 2024-03-25T18:16:55.563 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.561+0000 7f838affd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:55.563 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.561+0000 7f838affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:55.563 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.561+0000 7f838affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:55.563 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.561+0000 7f838affd640 10 monclient(hunting): my global_id is 4618 2024-03-25T18:16:55.563 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.561+0000 7f838affd640 1 -- 172.21.15.156:0/271982395 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8374004eb0 con 0x7f839413f710 2024-03-25T18:16:55.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.564+0000 7f838affd640 1 -- 172.21.15.156:0/271982395 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2766454087 0 0) 0x7f839413afc0 con 0x7f83940a98f0 2024-03-25T18:16:55.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.564+0000 7f838affd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:55.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.564+0000 7f838affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:55.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.564+0000 7f838affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:55.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.564+0000 7f838affd640 10 monclient(hunting): my global_id is 4622 2024-03-25T18:16:55.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.564+0000 7f838affd640 1 -- 172.21.15.156:0/271982395 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83940b71e0 con 0x7f83940a98f0 2024-03-25T18:16:55.573 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.571+0000 7f838affd640 1 -- 172.21.15.156:0/271982395 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3501075899 0 0) 0x7f8374004eb0 con 0x7f839413f710 2024-03-25T18:16:55.573 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.571+0000 7f838affd640 1 -- 172.21.15.156:0/271982395 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f839413afc0 con 0x7f839413f710 2024-03-25T18:16:55.573 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.571+0000 7f838affd640 1 -- 172.21.15.156:0/271982395 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f837c001a30 con 0x7f839413f710 2024-03-25T18:16:55.573 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.571+0000 7f838affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:55.573 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.571+0000 7f838affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:55.573 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.571+0000 7f838affd640 10 monclient(hunting): dump: 2024-03-25T18:16:55.573 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:55.573 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:55.573 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:55.573 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:55.573 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:55.573 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:55.573 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:55.574 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:55.574 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:55.574 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:55.574 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.571+0000 7f838affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:55.574 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.571+0000 7f838affd640 1 -- 172.21.15.156:0/271982395 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3093670906 0 0) 0x7f839413cc30 con 0x7f83941430c0 2024-03-25T18:16:55.574 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.571+0000 7f838affd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:55.574 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.571+0000 7f838affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:55.574 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.571+0000 7f838affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:55.574 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.571+0000 7f838affd640 10 monclient(hunting): my global_id is 4611 2024-03-25T18:16:55.574 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.571+0000 7f838affd640 1 -- 172.21.15.156:0/271982395 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8374004eb0 con 0x7f83941430c0 2024-03-25T18:16:55.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.578+0000 7f838affd640 1 -- 172.21.15.156:0/271982395 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4263375196 0 0) 0x7f839413afc0 con 0x7f839413f710 2024-03-25T18:16:55.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.578+0000 7f838affd640 1 -- 172.21.15.156:0/271982395 >> v1:172.21.15.156:6790/0 conn(0x7f83940a98f0 legacy=0x7f839413b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:55.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.578+0000 7f838affd640 1 -- 172.21.15.156:0/271982395 >> v1:172.21.15.156:6789/0 conn(0x7f83941430c0 legacy=0x7f83941e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:55.581 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.578+0000 7f838affd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:55.581 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.578+0000 7f838affd640 1 monclient: found mon.b 2024-03-25T18:16:55.581 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.578+0000 7f838affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:55.581 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.578+0000 7f838affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:16:55.581 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.578+0000 7f838affd640 1 -- 172.21.15.156:0/271982395 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839413b2a0 con 0x7f839413f710 2024-03-25T18:16:55.581 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.578+0000 7f838affd640 10 monclient: _finish_auth 0 2024-03-25T18:16:55.581 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.578+0000 7f838affd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:55.581 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.578+0000 7f838affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:55.581 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.578+0000 7f8398e0f640 5 monclient: authenticate success, global_id 4618 2024-03-25T18:16:55.581 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.578+0000 7f8398e0f640 10 monclient: _renew_subs 2024-03-25T18:16:55.581 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.578+0000 7f8398e0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:16:55.581 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.578+0000 7f8398e0f640 1 -- 172.21.15.156:0/271982395 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f839413b580 con 0x7f839413f710 2024-03-25T18:16:55.581 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.578+0000 7f8398e0f640 10 client.4618.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:55.581 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.578+0000 7f8398e0f640 10 monclient: _renew_subs 2024-03-25T18:16:55.581 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.578+0000 7f8398e0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:16:55.582 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.579+0000 7f8398e0f640 1 -- 172.21.15.156:0/271982395 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83940b7ca0 con 0x7f839413f710 2024-03-25T18:16:55.582 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.579+0000 7f8398e0f640 1 librados: init done 2024-03-25T18:16:55.582 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.579+0000 7f8398e0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:55.582 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.579+0000 7f8398e0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:55.582 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.579+0000 7f8398e0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:16:55.582 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.579+0000 7f8398e0f640 1 -- 172.21.15.156:0/271982395 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83940627b0 con 0x7f839413f710 2024-03-25T18:16:55.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.857+0000 7f838affd640 1 -- 172.21.15.156:0/271982395 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f837c001c70 con 0x7f839413f710 2024-03-25T18:16:55.858 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.857+0000 7f838affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:55.859 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.857+0000 7f838affd640 1 -- 172.21.15.156:0/271982395 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f837c003a60 con 0x7f839413f710 2024-03-25T18:16:55.859 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.857+0000 7f838affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:55.859 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.857+0000 7f838affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:55.859 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.857+0000 7f838affd640 10 monclient: dump: 2024-03-25T18:16:55.859 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:16:55.859 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:16:55.859 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:55.859 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:16:55.859 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:55.859 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:16:55.859 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:16:55.859 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:16:55.859 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:16:55.859 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:16:55.859 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.857+0000 7f839159a640 4 set_mon_vals no callback set 2024-03-25T18:16:55.860 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.858+0000 7f838affd640 1 -- 172.21.15.156:0/271982395 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f837c003c40 con 0x7f839413f710 2024-03-25T18:16:55.860 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.859+0000 7f839259c640 10 monclient: get_auth_request con 0x7f837406de00 auth_method 0 2024-03-25T18:16:55.860 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.859+0000 7f838affd640 10 client.4618.objecter ms_handle_connect 0x7f837406de00 2024-03-25T18:16:55.866 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.864+0000 7f838affd640 1 -- 172.21.15.156:0/271982395 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f837c0548c0 con 0x7f839413f710 2024-03-25T18:16:55.866 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.864+0000 7f838affd640 10 client.4618.objecter ms_dispatch 0x7f8394000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:55.866 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.864+0000 7f838affd640 3 client.4618.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:55.866 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.864+0000 7f838affd640 3 client.4618.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:55.866 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.864+0000 7f838affd640 20 client.4618.objecter dump_active .. 0 homeless 2024-03-25T18:16:55.866 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.865+0000 7f838affd640 1 -- 172.21.15.156:0/271982395 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f837c0503d0 con 0x7f839413f710 2024-03-25T18:16:55.866 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.865+0000 7f838affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:55.866 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:55.865+0000 7f838affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:56.121 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:56.120+0000 7f8398e0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:56.122 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:56.120+0000 7f8398e0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:56.122 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:56.120+0000 7f8398e0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:16:56.122 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:56.120+0000 7f8398e0f640 1 -- 172.21.15.156:0/271982395 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83940b7390 con 0x7f839413f710 2024-03-25T18:16:56.148 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:56.147+0000 7f838affd640 1 -- 172.21.15.156:0/271982395 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f83940b7390 con 0x7f839413f710 2024-03-25T18:16:56.148 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:56.147+0000 7f838affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:56.148 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:56.147+0000 7f838affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:56.148 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:16:56.148 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:56.151 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:56.150+0000 7f8398e0f640 10 librados: watch_flush enter 2024-03-25T18:16:56.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:56.150+0000 7f8398e0f640 10 librados: watch_flush exit 2024-03-25T18:16:56.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:56.150+0000 7f8398e0f640 20 client.4618.objecter shutdown clearing up homeless session... 2024-03-25T18:16:56.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:56.150+0000 7f8398e0f640 10 client.4618.objecter successfully canceled tick 2024-03-25T18:16:56.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:56.150+0000 7f8398e0f640 1 -- 172.21.15.156:0/271982395 >> v1:172.21.15.194:6800/2946427713 conn(0x7f837406de00 legacy=0x7f83740702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:56.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:56.150+0000 7f8398e0f640 10 monclient: shutdown 2024-03-25T18:16:56.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:56.150+0000 7f8398e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:56.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:56.150+0000 7f8398e0f640 1 -- 172.21.15.156:0/271982395 >> v1:172.21.15.194:6789/0 conn(0x7f839413f710 legacy=0x7f8394139e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:56.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:56.150+0000 7f8398e0f640 1 -- 172.21.15.156:0/271982395 shutdown_connections 2024-03-25T18:16:56.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:56.150+0000 7f8398e0f640 1 -- 172.21.15.156:0/271982395 >> 172.21.15.156:0/271982395 conn(0x7f83940b7ea0 msgr2=0x7f8394074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:56.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:56.150+0000 7f8398e0f640 1 -- 172.21.15.156:0/271982395 shutdown_connections 2024-03-25T18:16:56.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:56.150+0000 7f8398e0f640 1 -- 172.21.15.156:0/271982395 wait complete. 2024-03-25T18:16:56.152 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:16:56.150+0000 7f8398e0f640 1 librados: shutdown 2024-03-25T18:16:56.163 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:59.877 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:09:05.866 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:59.893 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:09:05.883 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:59.924 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:09:05.914 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:59.939 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:09:05.929 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:59.946 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:09:05.912 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:59.948 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:09:05.939 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:59.952 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:09:05.919 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:59.977 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:09:05.967 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:59.981 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:09:05.947 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:02.969 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:03.082 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.081+0000 7f776e724640 10 monclient: get_monmap_and_config 2024-03-25T18:17:03.082 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.081+0000 7f776e724640 10 monclient: build_initial_monmap 2024-03-25T18:17:03.082 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.081+0000 7f776e724640 10 monclient: monmap: 2024-03-25T18:17:03.083 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:17:03.083 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:03.083 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:17:03.082449+0000 2024-03-25T18:17:03.083 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:17:03.082449+0000 2024-03-25T18:17:03.083 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:03.083 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:03.083 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:17:03.083 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:17:03.083 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:17:03.083 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:03.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.082+0000 7f776e724640 1 Processor -- start 2024-03-25T18:17:03.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.082+0000 7f776e724640 1 -- start start 2024-03-25T18:17:03.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.082+0000 7f776e724640 10 monclient: init 2024-03-25T18:17:03.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.083+0000 7f776e724640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:03.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.083+0000 7f776e724640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:17:03.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.083+0000 7f776e724640 10 monclient(hunting): picked mon.noname-a con 0x7f77680b4d60 addr v1:172.21.15.156:6789/0 2024-03-25T18:17:03.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.083+0000 7f776e724640 10 monclient(hunting): picked mon.noname-c con 0x7f77680b1730 addr v1:172.21.15.194:6789/0 2024-03-25T18:17:03.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.083+0000 7f776e724640 10 monclient(hunting): picked mon.noname-b con 0x7f776814be90 addr v1:172.21.15.156:6790/0 2024-03-25T18:17:03.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.083+0000 7f776e724640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77680af940 con 0x7f77680b4d60 2024-03-25T18:17:03.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.083+0000 7f776e724640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77680b0a70 con 0x7f77680b1730 2024-03-25T18:17:03.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.083+0000 7f776e724640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77680b0d50 con 0x7f776814be90 2024-03-25T18:17:03.085 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.083+0000 7f776e724640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:03.086 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.083+0000 7f776e724640 10 monclient(hunting): authenticate will time out at 7504.393908s 2024-03-25T18:17:03.087 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.086+0000 7f7767fff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f77680b4d60 0x7f776813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:43796/0 (socket says 172.21.15.156:43796) 2024-03-25T18:17:03.087 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.086+0000 7f7767fff640 1 -- 172.21.15.156:0/4208999313 learned_addr learned my addr 172.21.15.156:0/4208999313 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:17:03.093 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.092+0000 7f7766ffd640 1 -- 172.21.15.156:0/4208999313 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1052944624 0 0) 0x7f77680af940 con 0x7f77680b4d60 2024-03-25T18:17:03.094 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.092+0000 7f7766ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:03.094 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.092+0000 7f7766ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:03.094 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.092+0000 7f7766ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:03.094 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.092+0000 7f7766ffd640 10 monclient(hunting): my global_id is 4614 2024-03-25T18:17:03.094 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.092+0000 7f7766ffd640 1 -- 172.21.15.156:0/4208999313 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7750003520 con 0x7f77680b4d60 2024-03-25T18:17:03.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.100+0000 7f7766ffd640 1 -- 172.21.15.156:0/4208999313 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2691273615 0 0) 0x7f7750003520 con 0x7f77680b4d60 2024-03-25T18:17:03.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.100+0000 7f7766ffd640 1 -- 172.21.15.156:0/4208999313 >> v1:172.21.15.156:6790/0 conn(0x7f776814be90 legacy=0x7f776814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:03.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.100+0000 7f7766ffd640 1 -- 172.21.15.156:0/4208999313 >> v1:172.21.15.194:6789/0 conn(0x7f77680b1730 legacy=0x7f77680b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:03.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.100+0000 7f7766ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:03.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.100+0000 7f7766ffd640 1 monclient: found mon.noname-a 2024-03-25T18:17:03.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.100+0000 7f7766ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:03.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.100+0000 7f7766ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:17:03.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.100+0000 7f7766ffd640 1 -- 172.21.15.156:0/4208999313 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77680b1030 con 0x7f77680b4d60 2024-03-25T18:17:03.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.100+0000 7f7766ffd640 10 monclient: _finish_auth 0 2024-03-25T18:17:03.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.100+0000 7f7766ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:03.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.100+0000 7f7766ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:03.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.100+0000 7f7766ffd640 1 -- 172.21.15.156:0/4208999313 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7758001690 con 0x7f77680b4d60 2024-03-25T18:17:03.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.100+0000 7f7766ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:03.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.100+0000 7f7766ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:03.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.100+0000 7f7766ffd640 10 monclient: dump: 2024-03-25T18:17:03.102 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:03.102 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:03.103 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:03.103 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:03.103 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:03.103 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:03.103 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:03.103 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:03.103 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:03.103 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:03.103 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.100+0000 7f776e724640 5 monclient: authenticate success, global_id 4614 2024-03-25T18:17:03.103 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.100+0000 7f776e724640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:03.110 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.109+0000 7f7766ffd640 1 -- 172.21.15.156:0/4208999313 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77580017f0 con 0x7f77680b4d60 2024-03-25T18:17:03.111 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.109+0000 7f7766ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:03.111 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.109+0000 7f7766ffd640 1 -- 172.21.15.156:0/4208999313 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7758001c00 con 0x7f77680b4d60 2024-03-25T18:17:03.111 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.109+0000 7f776e724640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:03.111 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.109+0000 7f776e724640 4 set_mon_vals no callback set 2024-03-25T18:17:03.111 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.109+0000 7f7766ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:03.111 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.109+0000 7f7766ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:03.111 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.109+0000 7f7766ffd640 10 monclient: dump: 2024-03-25T18:17:03.111 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:03.111 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:03.111 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:03.111 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:03.111 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:03.111 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:03.111 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:03.111 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:03.112 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:03.112 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:03.113 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.110+0000 7f776e724640 10 monclient: shutdown 2024-03-25T18:17:03.113 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.110+0000 7f776e724640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:03.113 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.110+0000 7f776e724640 1 -- 172.21.15.156:0/4208999313 >> v1:172.21.15.156:6789/0 conn(0x7f77680b4d60 legacy=0x7f776813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:03.113 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.110+0000 7f776e724640 1 -- 172.21.15.156:0/4208999313 shutdown_connections 2024-03-25T18:17:03.113 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.110+0000 7f776e724640 1 -- 172.21.15.156:0/4208999313 >> 172.21.15.156:0/4208999313 conn(0x7f77680617a0 msgr2=0x7f7768061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:03.113 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.110+0000 7f776e724640 1 -- 172.21.15.156:0/4208999313 shutdown_connections 2024-03-25T18:17:03.113 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.110+0000 7f776e724640 1 -- 172.21.15.156:0/4208999313 wait complete. 2024-03-25T18:17:03.113 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.111+0000 7f776e724640 10 monclient: build_initial_monmap 2024-03-25T18:17:03.113 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.111+0000 7f776e724640 10 monclient: monmap: 2024-03-25T18:17:03.113 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:17:03.113 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:03.113 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:17:03.114 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:17:03.114 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:03.114 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:17:03.114 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:17:03.114 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:17:03.114 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:17:03.114 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:03.114 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.111+0000 7f776e724640 1 librados: starting msgr at 2024-03-25T18:17:03.114 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.111+0000 7f776e724640 1 librados: starting objecter 2024-03-25T18:17:03.114 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.111+0000 7f776e724640 1 Processor -- start 2024-03-25T18:17:03.114 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.112+0000 7f776e724640 1 -- start start 2024-03-25T18:17:03.114 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.112+0000 7f776e724640 1 librados: setting wanted keys 2024-03-25T18:17:03.114 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.112+0000 7f776e724640 1 librados: calling monclient init 2024-03-25T18:17:03.114 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.112+0000 7f776e724640 10 monclient: init 2024-03-25T18:17:03.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.112+0000 7f776e724640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:03.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.112+0000 7f776e724640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:17:03.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.112+0000 7f776e724640 10 monclient(hunting): picked mon.noname-c con 0x7f77680b1730 addr v1:172.21.15.156:6790/0 2024-03-25T18:17:03.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.112+0000 7f776e724640 10 monclient(hunting): picked mon.noname-a con 0x7f77680b4d60 addr v1:172.21.15.156:6789/0 2024-03-25T18:17:03.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.112+0000 7f776e724640 10 monclient(hunting): picked mon.noname-b con 0x7f776814be90 addr v1:172.21.15.194:6789/0 2024-03-25T18:17:03.116 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.112+0000 7f776e724640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77681368f0 con 0x7f77680b4d60 2024-03-25T18:17:03.116 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.113+0000 7f776e724640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7768137750 con 0x7f776814be90 2024-03-25T18:17:03.116 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.113+0000 7f776e724640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77681363d0 con 0x7f77680b1730 2024-03-25T18:17:03.116 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.113+0000 7f776e724640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:03.116 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.113+0000 7f776e724640 10 monclient(hunting): authenticate will time out at 7504.423844s 2024-03-25T18:17:03.126 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.125+0000 7f776ccc1640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f776814be90 0x7f77681d7830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:51276/0 (socket says 172.21.15.156:51276) 2024-03-25T18:17:03.126 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.125+0000 7f776ccc1640 1 -- 172.21.15.156:0/2998506800 learned_addr learned my addr 172.21.15.156:0/2998506800 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:17:03.127 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.127+0000 7f7764ff9640 10 client.?.objecter ms_handle_connect 0x7f77680b1730 2024-03-25T18:17:03.127 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.127+0000 7f7764ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:03.131 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.131+0000 7f7764ff9640 10 client.?.objecter ms_handle_connect 0x7f776814be90 2024-03-25T18:17:03.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.131+0000 7f7764ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:03.133 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.133+0000 7f7764ff9640 10 client.?.objecter ms_handle_connect 0x7f77680b4d60 2024-03-25T18:17:03.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.133+0000 7f7764ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:03.142 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.141+0000 7f7764ff9640 1 -- 172.21.15.156:0/2998506800 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432487201 0 0) 0x7f77681363d0 con 0x7f77680b1730 2024-03-25T18:17:03.142 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.141+0000 7f7764ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:03.142 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.141+0000 7f7764ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:03.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.141+0000 7f7764ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:03.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.141+0000 7f7764ff9640 10 monclient(hunting): my global_id is 4628 2024-03-25T18:17:03.143 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.141+0000 7f7764ff9640 1 -- 172.21.15.156:0/2998506800 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7754004eb0 con 0x7f77680b1730 2024-03-25T18:17:03.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.143+0000 7f7764ff9640 1 -- 172.21.15.156:0/2998506800 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881920815 0 0) 0x7f7768137750 con 0x7f776814be90 2024-03-25T18:17:03.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.143+0000 7f7764ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:03.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.143+0000 7f7764ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:03.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.143+0000 7f7764ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:03.144 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.143+0000 7f7764ff9640 10 monclient(hunting): my global_id is 4624 2024-03-25T18:17:03.145 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.143+0000 7f7764ff9640 1 -- 172.21.15.156:0/2998506800 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77681363d0 con 0x7f776814be90 2024-03-25T18:17:03.147 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.146+0000 7f7764ff9640 1 -- 172.21.15.156:0/2998506800 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 345318377 0 0) 0x7f77681368f0 con 0x7f77680b4d60 2024-03-25T18:17:03.147 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.146+0000 7f7764ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:03.147 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.146+0000 7f7764ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:03.147 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.146+0000 7f7764ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:03.147 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.146+0000 7f7764ff9640 10 monclient(hunting): my global_id is 4617 2024-03-25T18:17:03.147 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.146+0000 7f7764ff9640 1 -- 172.21.15.156:0/2998506800 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7768137750 con 0x7f77680b4d60 2024-03-25T18:17:03.161 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.158+0000 7f7764ff9640 1 -- 172.21.15.156:0/2998506800 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1193237644 0 0) 0x7f7754004eb0 con 0x7f77680b1730 2024-03-25T18:17:03.161 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.158+0000 7f7764ff9640 1 -- 172.21.15.156:0/2998506800 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77681368f0 con 0x7f77680b1730 2024-03-25T18:17:03.161 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.158+0000 7f7764ff9640 1 -- 172.21.15.156:0/2998506800 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7758001bc0 con 0x7f77680b1730 2024-03-25T18:17:03.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.158+0000 7f7764ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:03.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.158+0000 7f7764ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:03.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.158+0000 7f7764ff9640 10 monclient(hunting): dump: 2024-03-25T18:17:03.162 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:03.162 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:03.162 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:03.162 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:03.162 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:03.162 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:03.162 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:03.162 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:03.162 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:03.162 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:03.162 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.158+0000 7f7764ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:03.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.161+0000 7f7764ff9640 1 -- 172.21.15.156:0/2998506800 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 814452727 0 0) 0x7f77681363d0 con 0x7f776814be90 2024-03-25T18:17:03.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.161+0000 7f7764ff9640 1 -- 172.21.15.156:0/2998506800 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7754004eb0 con 0x7f776814be90 2024-03-25T18:17:03.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.161+0000 7f7764ff9640 1 -- 172.21.15.156:0/2998506800 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7760001c10 con 0x7f776814be90 2024-03-25T18:17:03.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.161+0000 7f7764ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:03.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.161+0000 7f7764ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:03.163 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.161+0000 7f7764ff9640 10 monclient(hunting): dump: 2024-03-25T18:17:03.163 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:03.163 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:03.163 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:03.164 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:03.164 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:03.164 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:03.164 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:03.164 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:03.164 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:03.164 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:03.166 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.164+0000 7f7764ff9640 1 -- 172.21.15.156:0/2998506800 <== mon.2 v1:172.21.15.156:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3705279741 0 0) 0x7f77681368f0 con 0x7f77680b1730 2024-03-25T18:17:03.166 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.164+0000 7f7764ff9640 1 -- 172.21.15.156:0/2998506800 >> v1:172.21.15.194:6789/0 conn(0x7f776814be90 legacy=0x7f77681d7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:03.166 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.164+0000 7f7764ff9640 1 -- 172.21.15.156:0/2998506800 >> v1:172.21.15.156:6789/0 conn(0x7f77680b4d60 legacy=0x7f776812fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:03.166 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.164+0000 7f7764ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:17:03.166 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.164+0000 7f7764ff9640 1 monclient: found mon.c 2024-03-25T18:17:03.166 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.164+0000 7f7764ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:03.166 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.164+0000 7f7764ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:17:03.166 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.164+0000 7f7764ff9640 1 -- 172.21.15.156:0/2998506800 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7768130470 con 0x7f77680b1730 2024-03-25T18:17:03.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.164+0000 7f7764ff9640 10 monclient: _finish_auth 0 2024-03-25T18:17:03.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.164+0000 7f7764ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:17:03.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.164+0000 7f7764ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:03.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.164+0000 7f776e724640 5 monclient: authenticate success, global_id 4628 2024-03-25T18:17:03.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.164+0000 7f776e724640 10 monclient: _renew_subs 2024-03-25T18:17:03.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.164+0000 7f776e724640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:17:03.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.165+0000 7f776e724640 1 -- 172.21.15.156:0/2998506800 --> v1:172.21.15.156:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7768130720 con 0x7f77680b1730 2024-03-25T18:17:03.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.165+0000 7f776e724640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:03.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.165+0000 7f776e724640 10 monclient: _renew_subs 2024-03-25T18:17:03.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.165+0000 7f776e724640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:17:03.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.165+0000 7f776e724640 1 -- 172.21.15.156:0/2998506800 --> v1:172.21.15.156:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77681322f0 con 0x7f77680b1730 2024-03-25T18:17:03.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.165+0000 7f776e724640 1 librados: init done 2024-03-25T18:17:03.168 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.166+0000 7f776e724640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:03.168 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.166+0000 7f776e724640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:03.168 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.166+0000 7f776e724640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:17:03.168 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.166+0000 7f776e724640 1 -- 172.21.15.156:0/2998506800 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7768136210 con 0x7f77680b1730 2024-03-25T18:17:03.473 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.472+0000 7f7764ff9640 1 -- 172.21.15.156:0/2998506800 <== mon.2 v1:172.21.15.156:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7758004a90 con 0x7f77680b1730 2024-03-25T18:17:03.474 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.472+0000 7f7764ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:03.474 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.472+0000 7f7764ff9640 1 -- 172.21.15.156:0/2998506800 <== mon.2 v1:172.21.15.156:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7758004da0 con 0x7f77680b1730 2024-03-25T18:17:03.474 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.472+0000 7f7764ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:03.474 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.472+0000 7f7764ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:03.474 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.472+0000 7f7764ff9640 10 monclient: dump: 2024-03-25T18:17:03.474 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:03.474 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:03.474 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:03.474 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:03.474 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:03.474 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:03.474 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:03.474 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:03.475 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:03.475 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:03.475 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.472+0000 7f77667fc640 4 set_mon_vals no callback set 2024-03-25T18:17:03.475 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.473+0000 7f7764ff9640 1 -- 172.21.15.156:0/2998506800 <== mon.2 v1:172.21.15.156:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f775801ad70 con 0x7f77680b1730 2024-03-25T18:17:03.475 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.473+0000 7f77677fe640 10 monclient: get_auth_request con 0x7f775406df70 auth_method 0 2024-03-25T18:17:03.475 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.473+0000 7f7764ff9640 10 client.4628.objecter ms_handle_connect 0x7f775406df70 2024-03-25T18:17:03.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.480+0000 7f7764ff9640 1 -- 172.21.15.156:0/2998506800 <== mon.2 v1:172.21.15.156:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f7758054930 con 0x7f77680b1730 2024-03-25T18:17:03.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.480+0000 7f7764ff9640 10 client.4628.objecter ms_dispatch 0x7f7768000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:03.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.480+0000 7f7764ff9640 3 client.4628.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:03.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.480+0000 7f7764ff9640 3 client.4628.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:03.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.480+0000 7f7764ff9640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-25T18:17:03.482 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.480+0000 7f7764ff9640 1 -- 172.21.15.156:0/2998506800 <== mon.2 v1:172.21.15.156:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7758050270 con 0x7f77680b1730 2024-03-25T18:17:03.482 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.480+0000 7f7764ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:03.482 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.480+0000 7f7764ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:03.746 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.744+0000 7f776e724640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:03.746 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.744+0000 7f776e724640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:03.746 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.744+0000 7f776e724640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:17:03.746 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.744+0000 7f776e724640 1 -- 172.21.15.156:0/2998506800 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7768136560 con 0x7f77680b1730 2024-03-25T18:17:03.802 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.801+0000 7f7764ff9640 1 -- 172.21.15.156:0/2998506800 <== mon.2 v1:172.21.15.156:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f7758052370 con 0x7f77680b1730 2024-03-25T18:17:03.802 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.801+0000 7f7764ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:03.802 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.801+0000 7f7764ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:03.802 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:17:03.802 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:03.805 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.803+0000 7f776e724640 10 librados: watch_flush enter 2024-03-25T18:17:03.805 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.803+0000 7f776e724640 10 librados: watch_flush exit 2024-03-25T18:17:03.805 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.803+0000 7f776e724640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-25T18:17:03.805 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.803+0000 7f776e724640 10 client.4628.objecter successfully canceled tick 2024-03-25T18:17:03.805 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.804+0000 7f776e724640 1 -- 172.21.15.156:0/2998506800 >> v1:172.21.15.194:6800/2946427713 conn(0x7f775406df70 legacy=0x7f7754070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:03.805 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.804+0000 7f776e724640 10 monclient: shutdown 2024-03-25T18:17:03.806 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.804+0000 7f776e724640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:03.806 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.804+0000 7f776e724640 1 -- 172.21.15.156:0/2998506800 >> v1:172.21.15.156:6790/0 conn(0x7f77680b1730 legacy=0x7f7768135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:03.806 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.804+0000 7f776e724640 1 -- 172.21.15.156:0/2998506800 shutdown_connections 2024-03-25T18:17:03.806 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.804+0000 7f776e724640 1 -- 172.21.15.156:0/2998506800 >> 172.21.15.156:0/2998506800 conn(0x7f77680617a0 msgr2=0x7f7768138500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:03.806 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.804+0000 7f776e724640 1 -- 172.21.15.156:0/2998506800 shutdown_connections 2024-03-25T18:17:03.806 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.804+0000 7f776e724640 1 -- 172.21.15.156:0/2998506800 wait complete. 2024-03-25T18:17:03.806 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:03.804+0000 7f776e724640 1 librados: shutdown 2024-03-25T18:17:03.817 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:10.624 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:10.737 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.736+0000 7fbe87f15640 10 monclient: get_monmap_and_config 2024-03-25T18:17:10.738 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.736+0000 7fbe87f15640 10 monclient: build_initial_monmap 2024-03-25T18:17:10.738 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.736+0000 7fbe87f15640 10 monclient: monmap: 2024-03-25T18:17:10.738 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:17:10.738 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:10.738 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:17:10.737951+0000 2024-03-25T18:17:10.738 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:17:10.737951+0000 2024-03-25T18:17:10.738 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:10.738 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:10.738 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:17:10.738 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:17:10.738 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:17:10.739 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:10.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.737+0000 7fbe87f15640 1 Processor -- start 2024-03-25T18:17:10.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.737+0000 7fbe87f15640 1 -- start start 2024-03-25T18:17:10.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.737+0000 7fbe87f15640 10 monclient: init 2024-03-25T18:17:10.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.738+0000 7fbe87f15640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:10.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.738+0000 7fbe87f15640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:10.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.738+0000 7fbe87f15640 10 monclient(hunting): picked mon.noname-b con 0x7fbe800b4d60 addr v1:172.21.15.156:6790/0 2024-03-25T18:17:10.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.738+0000 7fbe87f15640 10 monclient(hunting): picked mon.noname-c con 0x7fbe800b1730 addr v1:172.21.15.194:6789/0 2024-03-25T18:17:10.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.738+0000 7fbe87f15640 10 monclient(hunting): picked mon.noname-a con 0x7fbe8014be90 addr v1:172.21.15.156:6789/0 2024-03-25T18:17:10.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.738+0000 7fbe87f15640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe800af940 con 0x7fbe8014be90 2024-03-25T18:17:10.741 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.738+0000 7fbe87f15640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe800b0a70 con 0x7fbe800b1730 2024-03-25T18:17:10.741 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.738+0000 7fbe87f15640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe800b0d50 con 0x7fbe800b4d60 2024-03-25T18:17:10.741 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.738+0000 7fbe87f15640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:10.741 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.738+0000 7fbe87f15640 10 monclient(hunting): authenticate will time out at 7512.049639s 2024-03-25T18:17:10.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.744+0000 7fbe864b2640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fbe8014be90 0x7fbe8014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:48514/0 (socket says 172.21.15.156:48514) 2024-03-25T18:17:10.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.744+0000 7fbe864b2640 1 -- 172.21.15.156:0/1536085610 learned_addr learned my addr 172.21.15.156:0/1536085610 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:17:10.767 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.766+0000 7fbe84caf640 1 -- 172.21.15.156:0/1536085610 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455884650 0 0) 0x7fbe800af940 con 0x7fbe8014be90 2024-03-25T18:17:10.767 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.766+0000 7fbe84caf640 10 monclient(hunting): _negotiate 2024-03-25T18:17:10.767 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.766+0000 7fbe84caf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:10.767 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.766+0000 7fbe84caf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:10.767 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.766+0000 7fbe84caf640 10 monclient(hunting): my global_id is 4620 2024-03-25T18:17:10.767 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.766+0000 7fbe84caf640 1 -- 172.21.15.156:0/1536085610 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe6c003520 con 0x7fbe8014be90 2024-03-25T18:17:10.768 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.767+0000 7fbe84caf640 1 -- 172.21.15.156:0/1536085610 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177461925 0 0) 0x7fbe800b0a70 con 0x7fbe800b1730 2024-03-25T18:17:10.768 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.767+0000 7fbe84caf640 10 monclient(hunting): _negotiate 2024-03-25T18:17:10.768 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.767+0000 7fbe84caf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:10.768 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.767+0000 7fbe84caf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:10.768 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.767+0000 7fbe84caf640 10 monclient(hunting): my global_id is 4627 2024-03-25T18:17:10.768 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.767+0000 7fbe84caf640 1 -- 172.21.15.156:0/1536085610 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe800af940 con 0x7fbe800b1730 2024-03-25T18:17:10.781 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.780+0000 7fbe84caf640 1 -- 172.21.15.156:0/1536085610 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 889560490 0 0) 0x7fbe800af940 con 0x7fbe800b1730 2024-03-25T18:17:10.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.780+0000 7fbe84caf640 1 -- 172.21.15.156:0/1536085610 >> v1:172.21.15.156:6790/0 conn(0x7fbe800b4d60 legacy=0x7fbe8013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:10.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.780+0000 7fbe84caf640 1 -- 172.21.15.156:0/1536085610 >> v1:172.21.15.156:6789/0 conn(0x7fbe8014be90 legacy=0x7fbe8014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:10.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.780+0000 7fbe84caf640 10 monclient: _finish_hunting 0 2024-03-25T18:17:10.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.780+0000 7fbe84caf640 1 monclient: found mon.noname-c 2024-03-25T18:17:10.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.780+0000 7fbe84caf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:10.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.780+0000 7fbe84caf640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:17:10.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.780+0000 7fbe84caf640 1 -- 172.21.15.156:0/1536085610 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe800b1030 con 0x7fbe800b1730 2024-03-25T18:17:10.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.780+0000 7fbe84caf640 10 monclient: _finish_auth 0 2024-03-25T18:17:10.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.780+0000 7fbe84caf640 10 monclient: _check_auth_tickets 2024-03-25T18:17:10.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.780+0000 7fbe84caf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:10.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.780+0000 7fbe84caf640 1 -- 172.21.15.156:0/1536085610 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fbe70001670 con 0x7fbe800b1730 2024-03-25T18:17:10.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.780+0000 7fbe87f15640 5 monclient: authenticate success, global_id 4627 2024-03-25T18:17:10.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.780+0000 7fbe87f15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:10.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.780+0000 7fbe84caf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:10.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.780+0000 7fbe84caf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:10.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.780+0000 7fbe84caf640 10 monclient: dump: 2024-03-25T18:17:10.783 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:10.783 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:10.783 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:10.783 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:10.783 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:10.783 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:10.783 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:10.783 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:10.783 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:10.783 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:10.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.780+0000 7fbe87f15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:10.792 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.791+0000 7fbe84caf640 1 -- 172.21.15.156:0/1536085610 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe700017d0 con 0x7fbe800b1730 2024-03-25T18:17:10.792 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.791+0000 7fbe84caf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:10.792 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.791+0000 7fbe84caf640 1 -- 172.21.15.156:0/1536085610 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fbe70001ba0 con 0x7fbe800b1730 2024-03-25T18:17:10.792 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.791+0000 7fbe84caf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:10.792 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.791+0000 7fbe84caf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:10.793 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.791+0000 7fbe84caf640 10 monclient: dump: 2024-03-25T18:17:10.793 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:10.793 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:10.793 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:10.793 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:10.793 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:10.793 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:10.793 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:10.793 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:10.793 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:10.793 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:10.793 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.791+0000 7fbe87f15640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:10.793 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.791+0000 7fbe87f15640 4 set_mon_vals no callback set 2024-03-25T18:17:10.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.792+0000 7fbe87f15640 10 monclient: shutdown 2024-03-25T18:17:10.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.792+0000 7fbe87f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:10.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.792+0000 7fbe87f15640 1 -- 172.21.15.156:0/1536085610 >> v1:172.21.15.194:6789/0 conn(0x7fbe800b1730 legacy=0x7fbe800b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:10.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.792+0000 7fbe87f15640 1 -- 172.21.15.156:0/1536085610 shutdown_connections 2024-03-25T18:17:10.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.792+0000 7fbe87f15640 1 -- 172.21.15.156:0/1536085610 >> 172.21.15.156:0/1536085610 conn(0x7fbe800617a0 msgr2=0x7fbe80061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:10.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.792+0000 7fbe87f15640 1 -- 172.21.15.156:0/1536085610 shutdown_connections 2024-03-25T18:17:10.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.792+0000 7fbe87f15640 1 -- 172.21.15.156:0/1536085610 wait complete. 2024-03-25T18:17:10.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.793+0000 7fbe87f15640 10 monclient: build_initial_monmap 2024-03-25T18:17:10.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.793+0000 7fbe87f15640 10 monclient: monmap: 2024-03-25T18:17:10.796 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:17:10.796 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:10.796 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:17:10.796 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:17:10.796 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:10.796 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:17:10.796 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:17:10.796 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:17:10.796 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:17:10.796 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:10.796 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.793+0000 7fbe87f15640 1 librados: starting msgr at 2024-03-25T18:17:10.796 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.793+0000 7fbe87f15640 1 librados: starting objecter 2024-03-25T18:17:10.796 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.794+0000 7fbe87f15640 1 Processor -- start 2024-03-25T18:17:10.796 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.794+0000 7fbe87f15640 1 -- start start 2024-03-25T18:17:10.796 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.794+0000 7fbe87f15640 1 librados: setting wanted keys 2024-03-25T18:17:10.796 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.794+0000 7fbe87f15640 1 librados: calling monclient init 2024-03-25T18:17:10.797 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.794+0000 7fbe87f15640 10 monclient: init 2024-03-25T18:17:10.798 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.794+0000 7fbe87f15640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:10.798 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.794+0000 7fbe87f15640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:17:10.798 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.794+0000 7fbe87f15640 10 monclient(hunting): picked mon.noname-b con 0x7fbe800b1730 addr v1:172.21.15.194:6789/0 2024-03-25T18:17:10.798 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.794+0000 7fbe87f15640 10 monclient(hunting): picked mon.noname-c con 0x7fbe800b4d60 addr v1:172.21.15.156:6790/0 2024-03-25T18:17:10.798 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.794+0000 7fbe87f15640 10 monclient(hunting): picked mon.noname-a con 0x7fbe8014be90 addr v1:172.21.15.156:6789/0 2024-03-25T18:17:10.798 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.794+0000 7fbe87f15640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe801d7850 con 0x7fbe8014be90 2024-03-25T18:17:10.798 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.794+0000 7fbe87f15640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe801382f0 con 0x7fbe800b1730 2024-03-25T18:17:10.798 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.795+0000 7fbe87f15640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe801d83f0 con 0x7fbe800b4d60 2024-03-25T18:17:10.798 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.795+0000 7fbe87f15640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:10.798 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.795+0000 7fbe87f15640 10 monclient(hunting): authenticate will time out at 7512.105835s 2024-03-25T18:17:10.808 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.807+0000 7fbe854b0640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7fbe800b4d60 0x7fbe80136770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:37182/0 (socket says 172.21.15.156:37182) 2024-03-25T18:17:10.808 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.807+0000 7fbe854b0640 1 -- 172.21.15.156:0/3617929360 learned_addr learned my addr 172.21.15.156:0/3617929360 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:17:10.810 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.809+0000 7fbe767fc640 10 client.?.objecter ms_handle_connect 0x7fbe800b4d60 2024-03-25T18:17:10.810 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.809+0000 7fbe767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:10.814 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.813+0000 7fbe767fc640 10 client.?.objecter ms_handle_connect 0x7fbe800b1730 2024-03-25T18:17:10.814 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.813+0000 7fbe767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:10.815 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.814+0000 7fbe767fc640 10 client.?.objecter ms_handle_connect 0x7fbe8014be90 2024-03-25T18:17:10.815 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.814+0000 7fbe767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:10.827 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.826+0000 7fbe767fc640 1 -- 172.21.15.156:0/3617929360 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1883103543 0 0) 0x7fbe801d83f0 con 0x7fbe800b4d60 2024-03-25T18:17:10.828 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.826+0000 7fbe767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:10.828 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.826+0000 7fbe767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:10.828 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.826+0000 7fbe767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:10.828 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.826+0000 7fbe767fc640 10 monclient(hunting): my global_id is 4634 2024-03-25T18:17:10.828 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.826+0000 7fbe767fc640 1 -- 172.21.15.156:0/3617929360 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe64004eb0 con 0x7fbe800b4d60 2024-03-25T18:17:10.828 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.827+0000 7fbe767fc640 1 -- 172.21.15.156:0/3617929360 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999641780 0 0) 0x7fbe801382f0 con 0x7fbe800b1730 2024-03-25T18:17:10.828 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.827+0000 7fbe767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:10.828 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.827+0000 7fbe767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:10.828 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.827+0000 7fbe767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:10.828 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.827+0000 7fbe767fc640 10 monclient(hunting): my global_id is 4630 2024-03-25T18:17:10.828 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.827+0000 7fbe767fc640 1 -- 172.21.15.156:0/3617929360 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe801d83f0 con 0x7fbe800b1730 2024-03-25T18:17:10.835 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.833+0000 7fbe767fc640 1 -- 172.21.15.156:0/3617929360 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2034377191 0 0) 0x7fbe801d83f0 con 0x7fbe800b1730 2024-03-25T18:17:10.835 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.833+0000 7fbe767fc640 1 -- 172.21.15.156:0/3617929360 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe801382f0 con 0x7fbe800b1730 2024-03-25T18:17:10.835 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.833+0000 7fbe767fc640 1 -- 172.21.15.156:0/3617929360 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fbe7c0018b0 con 0x7fbe800b1730 2024-03-25T18:17:10.835 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.833+0000 7fbe767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:10.835 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.833+0000 7fbe767fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:10.835 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.833+0000 7fbe767fc640 10 monclient(hunting): dump: 2024-03-25T18:17:10.835 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:10.835 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:10.835 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:10.836 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:10.836 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:10.836 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:10.836 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:10.836 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:10.836 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:10.836 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:10.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.833+0000 7fbe767fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:10.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.834+0000 7fbe767fc640 1 -- 172.21.15.156:0/3617929360 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675634712 0 0) 0x7fbe801d7850 con 0x7fbe8014be90 2024-03-25T18:17:10.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.834+0000 7fbe767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:10.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.834+0000 7fbe767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:10.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.834+0000 7fbe767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:10.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.834+0000 7fbe767fc640 10 monclient(hunting): my global_id is 4623 2024-03-25T18:17:10.836 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.834+0000 7fbe767fc640 1 -- 172.21.15.156:0/3617929360 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe801d83f0 con 0x7fbe8014be90 2024-03-25T18:17:10.837 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.835+0000 7fbe767fc640 1 -- 172.21.15.156:0/3617929360 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2493337014 0 0) 0x7fbe64004eb0 con 0x7fbe800b4d60 2024-03-25T18:17:10.837 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.835+0000 7fbe767fc640 1 -- 172.21.15.156:0/3617929360 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe801d7850 con 0x7fbe800b4d60 2024-03-25T18:17:10.837 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.835+0000 7fbe767fc640 1 -- 172.21.15.156:0/3617929360 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fbe70001be0 con 0x7fbe800b4d60 2024-03-25T18:17:10.837 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.835+0000 7fbe767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:10.837 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.835+0000 7fbe767fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:10.837 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.835+0000 7fbe767fc640 10 monclient(hunting): dump: 2024-03-25T18:17:10.837 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:10.837 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:10.837 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:10.838 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:10.838 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:10.838 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:10.838 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:10.838 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:10.838 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:10.838 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:10.843 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.841+0000 7fbe767fc640 1 -- 172.21.15.156:0/3617929360 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3573427137 0 0) 0x7fbe801d83f0 con 0x7fbe8014be90 2024-03-25T18:17:10.843 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.841+0000 7fbe767fc640 1 -- 172.21.15.156:0/3617929360 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe64004eb0 con 0x7fbe8014be90 2024-03-25T18:17:10.843 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.841+0000 7fbe767fc640 1 -- 172.21.15.156:0/3617929360 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fbe78001a30 con 0x7fbe8014be90 2024-03-25T18:17:10.843 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.841+0000 7fbe767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:10.843 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.841+0000 7fbe767fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:10.843 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.841+0000 7fbe767fc640 10 monclient(hunting): dump: 2024-03-25T18:17:10.843 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:10.843 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:10.843 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:10.843 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:10.843 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:10.843 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:10.844 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:10.844 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:10.844 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:10.844 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:10.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.844+0000 7fbe767fc640 1 -- 172.21.15.156:0/3617929360 <== mon.2 v1:172.21.15.156:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2057907022 0 0) 0x7fbe801d7850 con 0x7fbe800b4d60 2024-03-25T18:17:10.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.844+0000 7fbe767fc640 1 -- 172.21.15.156:0/3617929360 >> v1:172.21.15.194:6789/0 conn(0x7fbe800b1730 legacy=0x7fbe801d7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:10.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.844+0000 7fbe767fc640 1 -- 172.21.15.156:0/3617929360 >> v1:172.21.15.156:6789/0 conn(0x7fbe8014be90 legacy=0x7fbe8012fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:10.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.844+0000 7fbe767fc640 10 monclient: _finish_hunting 0 2024-03-25T18:17:10.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.844+0000 7fbe767fc640 1 monclient: found mon.c 2024-03-25T18:17:10.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.844+0000 7fbe767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:10.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.844+0000 7fbe767fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:17:10.846 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.844+0000 7fbe767fc640 1 -- 172.21.15.156:0/3617929360 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe80135c90 con 0x7fbe800b4d60 2024-03-25T18:17:10.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.844+0000 7fbe767fc640 10 monclient: _finish_auth 0 2024-03-25T18:17:10.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.844+0000 7fbe767fc640 10 monclient: _check_auth_tickets 2024-03-25T18:17:10.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.844+0000 7fbe767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:10.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.844+0000 7fbe87f15640 5 monclient: authenticate success, global_id 4634 2024-03-25T18:17:10.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.844+0000 7fbe87f15640 10 monclient: _renew_subs 2024-03-25T18:17:10.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.845+0000 7fbe87f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:17:10.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.845+0000 7fbe87f15640 1 -- 172.21.15.156:0/3617929360 --> v1:172.21.15.156:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe80136e80 con 0x7fbe800b4d60 2024-03-25T18:17:10.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.845+0000 7fbe87f15640 10 client.4634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:10.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.845+0000 7fbe87f15640 10 monclient: _renew_subs 2024-03-25T18:17:10.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.845+0000 7fbe87f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:17:10.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.845+0000 7fbe87f15640 1 -- 172.21.15.156:0/3617929360 --> v1:172.21.15.156:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe80143ae0 con 0x7fbe800b4d60 2024-03-25T18:17:10.847 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.845+0000 7fbe87f15640 1 librados: init done 2024-03-25T18:17:10.848 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.846+0000 7fbe87f15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:10.848 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.846+0000 7fbe87f15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:10.848 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.846+0000 7fbe87f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:17:10.848 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:10.846+0000 7fbe87f15640 1 -- 172.21.15.156:0/3617929360 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe800627b0 con 0x7fbe800b4d60 2024-03-25T18:17:11.177 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.176+0000 7fbe767fc640 1 -- 172.21.15.156:0/3617929360 <== mon.2 v1:172.21.15.156:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe700038a0 con 0x7fbe800b4d60 2024-03-25T18:17:11.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.176+0000 7fbe767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:11.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.176+0000 7fbe767fc640 1 -- 172.21.15.156:0/3617929360 <== mon.2 v1:172.21.15.156:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fbe70003bf0 con 0x7fbe800b4d60 2024-03-25T18:17:11.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.176+0000 7fbe767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:11.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.176+0000 7fbe767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:11.178 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.176+0000 7fbe767fc640 10 monclient: dump: 2024-03-25T18:17:11.178 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:11.178 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:11.178 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:11.178 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:11.178 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:11.178 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:11.178 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:11.178 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:11.179 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:11.179 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:11.179 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.176+0000 7fbe77fff640 4 set_mon_vals no callback set 2024-03-25T18:17:11.179 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.177+0000 7fbe767fc640 1 -- 172.21.15.156:0/3617929360 <== mon.2 v1:172.21.15.156:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7fbe70003e10 con 0x7fbe800b4d60 2024-03-25T18:17:11.179 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.178+0000 7fbe85cb1640 10 monclient: get_auth_request con 0x7fbe6406e0f0 auth_method 0 2024-03-25T18:17:11.179 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.178+0000 7fbe767fc640 10 client.4634.objecter ms_handle_connect 0x7fbe6406e0f0 2024-03-25T18:17:11.183 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.182+0000 7fbe767fc640 1 -- 172.21.15.156:0/3617929360 <== mon.2 v1:172.21.15.156:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7fbe70054940 con 0x7fbe800b4d60 2024-03-25T18:17:11.183 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.182+0000 7fbe767fc640 10 client.4634.objecter ms_dispatch 0x7fbe80000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:11.183 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.182+0000 7fbe767fc640 3 client.4634.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:11.184 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.182+0000 7fbe767fc640 3 client.4634.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:11.184 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.182+0000 7fbe767fc640 20 client.4634.objecter dump_active .. 0 homeless 2024-03-25T18:17:11.184 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.182+0000 7fbe767fc640 1 -- 172.21.15.156:0/3617929360 <== mon.2 v1:172.21.15.156:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbe70050280 con 0x7fbe800b4d60 2024-03-25T18:17:11.184 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.182+0000 7fbe767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:11.184 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.182+0000 7fbe767fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:11.438 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.437+0000 7fbe87f15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:11.438 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.437+0000 7fbe87f15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:11.438 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.437+0000 7fbe87f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:17:11.438 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.437+0000 7fbe87f15640 1 -- 172.21.15.156:0/3617929360 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe80069f40 con 0x7fbe800b4d60 2024-03-25T18:17:11.489 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.488+0000 7fbe767fc640 1 -- 172.21.15.156:0/3617929360 <== mon.2 v1:172.21.15.156:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7fbe70052380 con 0x7fbe800b4d60 2024-03-25T18:17:11.489 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.488+0000 7fbe767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:11.489 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.488+0000 7fbe767fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:11.490 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:17:11.490 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:11.493 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.491+0000 7fbe87f15640 10 librados: watch_flush enter 2024-03-25T18:17:11.493 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.491+0000 7fbe87f15640 10 librados: watch_flush exit 2024-03-25T18:17:11.493 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.491+0000 7fbe87f15640 20 client.4634.objecter shutdown clearing up homeless session... 2024-03-25T18:17:11.493 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.491+0000 7fbe87f15640 10 client.4634.objecter successfully canceled tick 2024-03-25T18:17:11.493 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.491+0000 7fbe87f15640 1 -- 172.21.15.156:0/3617929360 >> v1:172.21.15.194:6800/2946427713 conn(0x7fbe6406e0f0 legacy=0x7fbe640705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:11.493 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.491+0000 7fbe87f15640 10 monclient: shutdown 2024-03-25T18:17:11.493 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.491+0000 7fbe87f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:11.493 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.491+0000 7fbe87f15640 1 -- 172.21.15.156:0/3617929360 >> v1:172.21.15.156:6790/0 conn(0x7fbe800b4d60 legacy=0x7fbe80136770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:11.493 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.491+0000 7fbe87f15640 1 -- 172.21.15.156:0/3617929360 shutdown_connections 2024-03-25T18:17:11.494 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.491+0000 7fbe87f15640 1 -- 172.21.15.156:0/3617929360 >> 172.21.15.156:0/3617929360 conn(0x7fbe800617a0 msgr2=0x7fbe8013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:11.494 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.491+0000 7fbe87f15640 1 -- 172.21.15.156:0/3617929360 shutdown_connections 2024-03-25T18:17:11.494 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.491+0000 7fbe87f15640 1 -- 172.21.15.156:0/3617929360 wait complete. 2024-03-25T18:17:11.494 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:11.491+0000 7fbe87f15640 1 librados: shutdown 2024-03-25T18:17:11.503 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:18.310 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:18.412 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.410+0000 7f176bd02640 10 monclient: get_monmap_and_config 2024-03-25T18:17:18.413 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.410+0000 7f176bd02640 10 monclient: build_initial_monmap 2024-03-25T18:17:18.413 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.410+0000 7f176bd02640 10 monclient: monmap: 2024-03-25T18:17:18.413 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:17:18.413 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:18.413 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:17:18.412509+0000 2024-03-25T18:17:18.413 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:17:18.412509+0000 2024-03-25T18:17:18.413 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:18.413 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:18.413 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:17:18.413 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:17:18.413 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:17:18.413 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:18.413 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.411+0000 7f176bd02640 1 Processor -- start 2024-03-25T18:17:18.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.411+0000 7f176bd02640 1 -- start start 2024-03-25T18:17:18.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.411+0000 7f176bd02640 10 monclient: init 2024-03-25T18:17:18.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.411+0000 7f176bd02640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:18.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.411+0000 7f176bd02640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:17:18.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.411+0000 7f176bd02640 10 monclient(hunting): picked mon.noname-c con 0x7f17640a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:17:18.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.411+0000 7f176bd02640 10 monclient(hunting): picked mon.noname-b con 0x7f176413f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:17:18.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.411+0000 7f176bd02640 10 monclient(hunting): picked mon.noname-a con 0x7f17641430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:17:18.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.411+0000 7f176bd02640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1764142520 con 0x7f17641430c0 2024-03-25T18:17:18.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.411+0000 7f176bd02640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17640b28a0 con 0x7f17640a98f0 2024-03-25T18:17:18.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.411+0000 7f176bd02640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17640b2a30 con 0x7f176413f710 2024-03-25T18:17:18.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.412+0000 7f176bd02640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:18.415 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.412+0000 7f176bd02640 10 monclient(hunting): authenticate will time out at 7519.723803s 2024-03-25T18:17:18.434 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.432+0000 7f176a29f640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f17641430c0 0x7f176414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:40518/0 (socket says 172.21.15.156:40518) 2024-03-25T18:17:18.434 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.432+0000 7f176a29f640 1 -- 172.21.15.156:0/149025004 learned_addr learned my addr 172.21.15.156:0/149025004 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:17:18.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.456+0000 7f1768a9c640 1 -- 172.21.15.156:0/149025004 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685738709 0 0) 0x7f1764142520 con 0x7f17641430c0 2024-03-25T18:17:18.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.456+0000 7f1768a9c640 10 monclient(hunting): _negotiate 2024-03-25T18:17:18.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.456+0000 7f1768a9c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:18.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.456+0000 7f1768a9c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:18.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.456+0000 7f1768a9c640 10 monclient(hunting): my global_id is 4626 2024-03-25T18:17:18.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.456+0000 7f1768a9c640 1 -- 172.21.15.156:0/149025004 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1750003520 con 0x7f17641430c0 2024-03-25T18:17:18.470 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.467+0000 7f1768a9c640 1 -- 172.21.15.156:0/149025004 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 915950155 0 0) 0x7f1750003520 con 0x7f17641430c0 2024-03-25T18:17:18.470 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.467+0000 7f1768a9c640 1 -- 172.21.15.156:0/149025004 >> v1:172.21.15.156:6790/0 conn(0x7f176413f710 legacy=0x7f17641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:18.470 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.467+0000 7f1768a9c640 1 -- 172.21.15.156:0/149025004 >> v1:172.21.15.194:6789/0 conn(0x7f17640a98f0 legacy=0x7f17640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:18.470 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.467+0000 7f1768a9c640 10 monclient: _finish_hunting 0 2024-03-25T18:17:18.470 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.467+0000 7f1768a9c640 1 monclient: found mon.noname-a 2024-03-25T18:17:18.470 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.467+0000 7f1768a9c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:18.470 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.467+0000 7f1768a9c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:17:18.470 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.467+0000 7f1768a9c640 1 -- 172.21.15.156:0/149025004 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17640b44c0 con 0x7f17641430c0 2024-03-25T18:17:18.470 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.467+0000 7f1768a9c640 10 monclient: _finish_auth 0 2024-03-25T18:17:18.470 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.467+0000 7f1768a9c640 10 monclient: _check_auth_tickets 2024-03-25T18:17:18.471 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.467+0000 7f1768a9c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:18.471 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.467+0000 7f1768a9c640 1 -- 172.21.15.156:0/149025004 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f175c001a10 con 0x7f17641430c0 2024-03-25T18:17:18.471 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.467+0000 7f176bd02640 5 monclient: authenticate success, global_id 4626 2024-03-25T18:17:18.471 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.467+0000 7f176bd02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:18.471 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.467+0000 7f1768a9c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:18.471 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.467+0000 7f1768a9c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:18.471 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.467+0000 7f1768a9c640 10 monclient: dump: 2024-03-25T18:17:18.471 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:18.471 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:18.471 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:18.471 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:18.471 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:18.471 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:18.471 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:18.471 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:18.471 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:18.472 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:18.472 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.467+0000 7f176bd02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:18.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.532+0000 7f1768a9c640 1 -- 172.21.15.156:0/149025004 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f175c001b70 con 0x7f17641430c0 2024-03-25T18:17:18.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.532+0000 7f1768a9c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:18.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.532+0000 7f1768a9c640 1 -- 172.21.15.156:0/149025004 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f175c001f80 con 0x7f17641430c0 2024-03-25T18:17:18.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.532+0000 7f1768a9c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:18.534 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.532+0000 7f1768a9c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:18.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.532+0000 7f1768a9c640 10 monclient: dump: 2024-03-25T18:17:18.535 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:18.535 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:18.535 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:18.535 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:18.535 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:18.535 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:18.535 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:18.535 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:18.535 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:18.535 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:18.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.532+0000 7f176bd02640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:18.535 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.532+0000 7f176bd02640 4 set_mon_vals no callback set 2024-03-25T18:17:18.537 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.533+0000 7f176bd02640 10 monclient: shutdown 2024-03-25T18:17:18.537 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.533+0000 7f176bd02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:18.537 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.533+0000 7f176bd02640 1 -- 172.21.15.156:0/149025004 >> v1:172.21.15.156:6789/0 conn(0x7f17641430c0 legacy=0x7f176414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:18.537 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.533+0000 7f176bd02640 1 -- 172.21.15.156:0/149025004 shutdown_connections 2024-03-25T18:17:18.537 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.533+0000 7f176bd02640 1 -- 172.21.15.156:0/149025004 >> 172.21.15.156:0/149025004 conn(0x7f17640b7ea0 msgr2=0x7f17640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:18.537 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.533+0000 7f176bd02640 1 -- 172.21.15.156:0/149025004 shutdown_connections 2024-03-25T18:17:18.537 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.533+0000 7f176bd02640 1 -- 172.21.15.156:0/149025004 wait complete. 2024-03-25T18:17:18.538 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.534+0000 7f176bd02640 10 monclient: build_initial_monmap 2024-03-25T18:17:18.538 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.534+0000 7f176bd02640 10 monclient: monmap: 2024-03-25T18:17:18.538 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:17:18.538 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:18.538 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:17:18.538 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:17:18.538 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:18.538 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:17:18.538 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:17:18.538 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:17:18.538 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:17:18.538 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:18.538 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.534+0000 7f176bd02640 1 librados: starting msgr at 2024-03-25T18:17:18.538 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.534+0000 7f176bd02640 1 librados: starting objecter 2024-03-25T18:17:18.539 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.534+0000 7f176bd02640 1 Processor -- start 2024-03-25T18:17:18.539 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.534+0000 7f176bd02640 1 -- start start 2024-03-25T18:17:18.539 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.534+0000 7f176bd02640 1 librados: setting wanted keys 2024-03-25T18:17:18.539 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.534+0000 7f176bd02640 1 librados: calling monclient init 2024-03-25T18:17:18.539 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.534+0000 7f176bd02640 10 monclient: init 2024-03-25T18:17:18.539 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.535+0000 7f176bd02640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:18.539 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.535+0000 7f176bd02640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:18.539 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.535+0000 7f176bd02640 10 monclient(hunting): picked mon.noname-b con 0x7f17640a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:17:18.539 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.535+0000 7f176bd02640 10 monclient(hunting): picked mon.noname-a con 0x7f176413f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:17:18.539 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.535+0000 7f176bd02640 10 monclient(hunting): picked mon.noname-c con 0x7f17641430c0 addr v1:172.21.15.156:6790/0 2024-03-25T18:17:18.539 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.535+0000 7f176bd02640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1764139a10 con 0x7f176413f710 2024-03-25T18:17:18.539 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.535+0000 7f176bd02640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17641d7880 con 0x7f17640a98f0 2024-03-25T18:17:18.539 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.535+0000 7f176bd02640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1764137890 con 0x7f17641430c0 2024-03-25T18:17:18.539 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.535+0000 7f176bd02640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:18.539 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.535+0000 7f176bd02640 10 monclient(hunting): authenticate will time out at 7519.847165s 2024-03-25T18:17:18.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.545+0000 7f176a29f640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f17641430c0 0x7f17641e1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:34464/0 (socket says 172.21.15.156:34464) 2024-03-25T18:17:18.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.545+0000 7f176a29f640 1 -- 172.21.15.156:0/1827245681 learned_addr learned my addr 172.21.15.156:0/1827245681 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:17:18.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.546+0000 7f175a7fc640 10 client.?.objecter ms_handle_connect 0x7f17641430c0 2024-03-25T18:17:18.548 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.546+0000 7f175a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:18.553 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.551+0000 7f175a7fc640 10 client.?.objecter ms_handle_connect 0x7f17640a98f0 2024-03-25T18:17:18.553 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.551+0000 7f175a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:18.559 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.557+0000 7f175a7fc640 10 client.?.objecter ms_handle_connect 0x7f176413f710 2024-03-25T18:17:18.559 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.557+0000 7f175a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:18.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.565+0000 7f175a7fc640 1 -- 172.21.15.156:0/1827245681 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951330091 0 0) 0x7f17641d7880 con 0x7f17640a98f0 2024-03-25T18:17:18.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.565+0000 7f175a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:18.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.565+0000 7f175a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:18.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.565+0000 7f175a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:18.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.565+0000 7f175a7fc640 10 monclient(hunting): my global_id is 4633 2024-03-25T18:17:18.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.565+0000 7f175a7fc640 1 -- 172.21.15.156:0/1827245681 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1748004eb0 con 0x7f17640a98f0 2024-03-25T18:17:18.568 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.566+0000 7f175a7fc640 1 -- 172.21.15.156:0/1827245681 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1575465137 0 0) 0x7f1764139a10 con 0x7f176413f710 2024-03-25T18:17:18.568 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.566+0000 7f175a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:18.568 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.566+0000 7f175a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:18.568 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.566+0000 7f175a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:18.568 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.566+0000 7f175a7fc640 10 monclient(hunting): my global_id is 4629 2024-03-25T18:17:18.568 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.566+0000 7f175a7fc640 1 -- 172.21.15.156:0/1827245681 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17641d7880 con 0x7f176413f710 2024-03-25T18:17:18.569 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.567+0000 7f175a7fc640 1 -- 172.21.15.156:0/1827245681 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901231172 0 0) 0x7f1764137890 con 0x7f17641430c0 2024-03-25T18:17:18.569 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.567+0000 7f175a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:18.569 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.567+0000 7f175a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:18.570 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.567+0000 7f175a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:18.570 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.567+0000 7f175a7fc640 10 monclient(hunting): my global_id is 4637 2024-03-25T18:17:18.570 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.567+0000 7f175a7fc640 1 -- 172.21.15.156:0/1827245681 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1764139a10 con 0x7f17641430c0 2024-03-25T18:17:18.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.576+0000 7f175a7fc640 1 -- 172.21.15.156:0/1827245681 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1832941447 0 0) 0x7f1764139a10 con 0x7f17641430c0 2024-03-25T18:17:18.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.576+0000 7f175a7fc640 1 -- 172.21.15.156:0/1827245681 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1764137890 con 0x7f17641430c0 2024-03-25T18:17:18.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.576+0000 7f175a7fc640 1 -- 172.21.15.156:0/1827245681 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f175c001e80 con 0x7f17641430c0 2024-03-25T18:17:18.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.576+0000 7f175a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:18.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.576+0000 7f175a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:18.579 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.576+0000 7f175a7fc640 10 monclient(hunting): dump: 2024-03-25T18:17:18.579 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:18.579 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:18.579 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:18.579 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:18.579 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:18.579 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:18.579 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:18.579 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:18.580 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:18.580 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:18.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.576+0000 7f175a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:18.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.576+0000 7f175a7fc640 1 -- 172.21.15.156:0/1827245681 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 660311193 0 0) 0x7f17641d7880 con 0x7f176413f710 2024-03-25T18:17:18.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.576+0000 7f175a7fc640 1 -- 172.21.15.156:0/1827245681 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1764139a10 con 0x7f176413f710 2024-03-25T18:17:18.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.576+0000 7f175a7fc640 1 -- 172.21.15.156:0/1827245681 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f17540018b0 con 0x7f176413f710 2024-03-25T18:17:18.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.576+0000 7f175a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:18.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.576+0000 7f175a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:18.580 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.576+0000 7f175a7fc640 10 monclient(hunting): dump: 2024-03-25T18:17:18.580 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:18.580 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:18.580 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:18.580 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:18.580 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:18.580 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:18.580 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:18.581 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:18.581 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:18.581 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:18.587 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.585+0000 7f175a7fc640 1 -- 172.21.15.156:0/1827245681 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2775527732 0 0) 0x7f1748004eb0 con 0x7f17640a98f0 2024-03-25T18:17:18.587 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.585+0000 7f175a7fc640 1 -- 172.21.15.156:0/1827245681 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17641d7880 con 0x7f17640a98f0 2024-03-25T18:17:18.587 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.585+0000 7f175a7fc640 1 -- 172.21.15.156:0/1827245681 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f17600018c0 con 0x7f17640a98f0 2024-03-25T18:17:18.587 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.585+0000 7f175a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:18.587 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.585+0000 7f175a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:18.587 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.585+0000 7f175a7fc640 10 monclient(hunting): dump: 2024-03-25T18:17:18.587 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:18.587 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:18.587 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:18.587 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:18.587 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:18.588 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:18.588 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:18.588 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:18.588 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:18.588 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:18.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.586+0000 7f175a7fc640 1 -- 172.21.15.156:0/1827245681 <== mon.2 v1:172.21.15.156:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2741409002 0 0) 0x7f1764137890 con 0x7f17641430c0 2024-03-25T18:17:18.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.586+0000 7f175a7fc640 1 -- 172.21.15.156:0/1827245681 >> v1:172.21.15.194:6789/0 conn(0x7f17640a98f0 legacy=0x7f17641d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:18.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.586+0000 7f175a7fc640 1 -- 172.21.15.156:0/1827245681 >> v1:172.21.15.156:6789/0 conn(0x7f176413f710 legacy=0x7f1764138370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:18.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.586+0000 7f175a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:17:18.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.586+0000 7f175a7fc640 1 monclient: found mon.c 2024-03-25T18:17:18.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.586+0000 7f175a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:18.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.586+0000 7f175a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:17:18.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.586+0000 7f175a7fc640 1 -- 172.21.15.156:0/1827245681 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17641d84b0 con 0x7f17641430c0 2024-03-25T18:17:18.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.586+0000 7f175a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:17:18.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.586+0000 7f175a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:17:18.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.586+0000 7f175a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:18.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.586+0000 7f176bd02640 5 monclient: authenticate success, global_id 4637 2024-03-25T18:17:18.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.586+0000 7f176bd02640 10 monclient: _renew_subs 2024-03-25T18:17:18.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.586+0000 7f176bd02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:17:18.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.586+0000 7f176bd02640 1 -- 172.21.15.156:0/1827245681 --> v1:172.21.15.156:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17641d51f0 con 0x7f17641430c0 2024-03-25T18:17:18.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.586+0000 7f176bd02640 10 client.4637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:18.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.586+0000 7f176bd02640 10 monclient: _renew_subs 2024-03-25T18:17:18.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.586+0000 7f176bd02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:17:18.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.586+0000 7f176bd02640 1 -- 172.21.15.156:0/1827245681 --> v1:172.21.15.156:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1764138c60 con 0x7f17641430c0 2024-03-25T18:17:18.590 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.586+0000 7f176bd02640 1 librados: init done 2024-03-25T18:17:18.591 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.587+0000 7f176bd02640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:18.591 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.587+0000 7f176bd02640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:18.591 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.587+0000 7f176bd02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:17:18.591 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.587+0000 7f176bd02640 1 -- 172.21.15.156:0/1827245681 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17641d7a10 con 0x7f17641430c0 2024-03-25T18:17:18.892 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.889+0000 7f175a7fc640 1 -- 172.21.15.156:0/1827245681 <== mon.2 v1:172.21.15.156:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f175c0020a0 con 0x7f17641430c0 2024-03-25T18:17:18.892 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.889+0000 7f175a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:18.892 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.889+0000 7f175a7fc640 1 -- 172.21.15.156:0/1827245681 <== mon.2 v1:172.21.15.156:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f175c004fb0 con 0x7f17641430c0 2024-03-25T18:17:18.892 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.889+0000 7f175a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:18.892 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.889+0000 7f175a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:18.892 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.889+0000 7f175a7fc640 10 monclient: dump: 2024-03-25T18:17:18.892 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:18.892 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:18.892 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:18.892 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:18.892 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:18.892 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:18.892 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:18.893 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:18.893 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:18.893 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:18.893 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.889+0000 7f175bfff640 4 set_mon_vals no callback set 2024-03-25T18:17:18.893 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.890+0000 7f175a7fc640 1 -- 172.21.15.156:0/1827245681 <== mon.2 v1:172.21.15.156:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f175c01af80 con 0x7f17641430c0 2024-03-25T18:17:18.893 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.891+0000 7f1769a9e640 10 monclient: get_auth_request con 0x7f174806e0b0 auth_method 0 2024-03-25T18:17:18.894 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.891+0000 7f175a7fc640 10 client.4637.objecter ms_handle_connect 0x7f174806e0b0 2024-03-25T18:17:18.901 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.897+0000 7f175a7fc640 1 -- 172.21.15.156:0/1827245681 <== mon.2 v1:172.21.15.156:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f175c054b20 con 0x7f17641430c0 2024-03-25T18:17:18.901 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.897+0000 7f175a7fc640 10 client.4637.objecter ms_dispatch 0x7f1764000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:18.902 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.897+0000 7f175a7fc640 3 client.4637.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:18.902 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.897+0000 7f175a7fc640 3 client.4637.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:18.902 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.898+0000 7f175a7fc640 20 client.4637.objecter dump_active .. 0 homeless 2024-03-25T18:17:18.902 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.898+0000 7f175a7fc640 1 -- 172.21.15.156:0/1827245681 <== mon.2 v1:172.21.15.156:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f175c050460 con 0x7f17641430c0 2024-03-25T18:17:18.902 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.898+0000 7f175a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:18.902 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:18.898+0000 7f175a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:19.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:19.150+0000 7f176bd02640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:19.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:19.150+0000 7f176bd02640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:19.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:19.150+0000 7f176bd02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:17:19.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:19.150+0000 7f176bd02640 1 -- 172.21.15.156:0/1827245681 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17640627b0 con 0x7f17641430c0 2024-03-25T18:17:19.191 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:19.189+0000 7f175a7fc640 1 -- 172.21.15.156:0/1827245681 <== mon.2 v1:172.21.15.156:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f175c052560 con 0x7f17641430c0 2024-03-25T18:17:19.191 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:19.189+0000 7f175a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:19.191 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:19.189+0000 7f175a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:19.191 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:17:19.192 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:19.194 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:19.192+0000 7f176bd02640 10 librados: watch_flush enter 2024-03-25T18:17:19.194 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:19.192+0000 7f176bd02640 10 librados: watch_flush exit 2024-03-25T18:17:19.195 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:19.192+0000 7f176bd02640 20 client.4637.objecter shutdown clearing up homeless session... 2024-03-25T18:17:19.195 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:19.192+0000 7f176bd02640 10 client.4637.objecter successfully canceled tick 2024-03-25T18:17:19.195 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:19.192+0000 7f176bd02640 1 -- 172.21.15.156:0/1827245681 >> v1:172.21.15.194:6800/2946427713 conn(0x7f174806e0b0 legacy=0x7f1748070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:19.195 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:19.192+0000 7f176bd02640 10 monclient: shutdown 2024-03-25T18:17:19.195 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:19.192+0000 7f176bd02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:19.195 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:19.192+0000 7f176bd02640 1 -- 172.21.15.156:0/1827245681 >> v1:172.21.15.156:6790/0 conn(0x7f17641430c0 legacy=0x7f17641e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:19.195 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:19.192+0000 7f176bd02640 1 -- 172.21.15.156:0/1827245681 shutdown_connections 2024-03-25T18:17:19.195 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:19.192+0000 7f176bd02640 1 -- 172.21.15.156:0/1827245681 >> 172.21.15.156:0/1827245681 conn(0x7f17640b7ea0 msgr2=0x7f1764074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:19.195 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:19.192+0000 7f176bd02640 1 -- 172.21.15.156:0/1827245681 shutdown_connections 2024-03-25T18:17:19.195 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:19.192+0000 7f176bd02640 1 -- 172.21.15.156:0/1827245681 wait complete. 2024-03-25T18:17:19.195 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:19.192+0000 7f176bd02640 1 librados: shutdown 2024-03-25T18:17:19.206 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:19.925 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:09:25.915 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:19.933 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:09:25.923 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:19.965 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:09:25.956 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:19.982 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:09:25.949 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:19.986 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:09:25.976 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:19.987 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:09:25.954 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:19.992 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:09:25.982 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:20.014 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:09:25.981 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:20.033 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:09:26.023 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:26.013 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:26.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.127+0000 7fb56bcc9640 10 monclient: get_monmap_and_config 2024-03-25T18:17:26.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.127+0000 7fb56bcc9640 10 monclient: build_initial_monmap 2024-03-25T18:17:26.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.127+0000 7fb56bcc9640 10 monclient: monmap: 2024-03-25T18:17:26.129 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:17:26.129 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:26.129 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:17:26.128884+0000 2024-03-25T18:17:26.129 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:17:26.128884+0000 2024-03-25T18:17:26.129 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:26.129 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:26.129 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:17:26.129 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:17:26.129 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:17:26.130 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:26.131 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.128+0000 7fb56bcc9640 1 Processor -- start 2024-03-25T18:17:26.131 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.128+0000 7fb56bcc9640 1 -- start start 2024-03-25T18:17:26.131 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.128+0000 7fb56bcc9640 10 monclient: init 2024-03-25T18:17:26.131 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.129+0000 7fb56bcc9640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:26.131 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.129+0000 7fb56bcc9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:26.131 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.129+0000 7fb56bcc9640 10 monclient(hunting): picked mon.noname-a con 0x7fb5640a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:17:26.131 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.129+0000 7fb56bcc9640 10 monclient(hunting): picked mon.noname-b con 0x7fb56413f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:17:26.131 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.129+0000 7fb56bcc9640 10 monclient(hunting): picked mon.noname-c con 0x7fb5641430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:17:26.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.129+0000 7fb56bcc9640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb564142520 con 0x7fb5640a98f0 2024-03-25T18:17:26.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.129+0000 7fb56bcc9640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5640b28a0 con 0x7fb5641430c0 2024-03-25T18:17:26.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.129+0000 7fb56bcc9640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5640b2a30 con 0x7fb56413f710 2024-03-25T18:17:26.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.129+0000 7fb56bcc9640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:26.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.129+0000 7fb56bcc9640 10 monclient(hunting): authenticate will time out at 7527.440630s 2024-03-25T18:17:26.141 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.140+0000 7fb56a266640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fb5641430c0 0x7fb56414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:44328/0 (socket says 172.21.15.156:44328) 2024-03-25T18:17:26.141 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.140+0000 7fb56a266640 1 -- 172.21.15.156:0/2226531033 learned_addr learned my addr 172.21.15.156:0/2226531033 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:17:26.157 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.156+0000 7fb568a63640 1 -- 172.21.15.156:0/2226531033 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178587172 0 0) 0x7fb5640b28a0 con 0x7fb5641430c0 2024-03-25T18:17:26.157 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.156+0000 7fb568a63640 10 monclient(hunting): _negotiate 2024-03-25T18:17:26.157 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.156+0000 7fb568a63640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:26.157 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.156+0000 7fb568a63640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:26.157 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.156+0000 7fb568a63640 10 monclient(hunting): my global_id is 4636 2024-03-25T18:17:26.157 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.156+0000 7fb568a63640 1 -- 172.21.15.156:0/2226531033 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb550003520 con 0x7fb5641430c0 2024-03-25T18:17:26.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.157+0000 7fb568a63640 1 -- 172.21.15.156:0/2226531033 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3491111996 0 0) 0x7fb564142520 con 0x7fb5640a98f0 2024-03-25T18:17:26.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.157+0000 7fb568a63640 10 monclient(hunting): _negotiate 2024-03-25T18:17:26.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.157+0000 7fb568a63640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:26.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.157+0000 7fb568a63640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:26.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.157+0000 7fb568a63640 10 monclient(hunting): my global_id is 4632 2024-03-25T18:17:26.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.157+0000 7fb568a63640 1 -- 172.21.15.156:0/2226531033 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5640b28a0 con 0x7fb5640a98f0 2024-03-25T18:17:26.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.171+0000 7fb568a63640 1 -- 172.21.15.156:0/2226531033 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3728727514 0 0) 0x7fb550003520 con 0x7fb5641430c0 2024-03-25T18:17:26.172 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.171+0000 7fb568a63640 1 -- 172.21.15.156:0/2226531033 >> v1:172.21.15.156:6790/0 conn(0x7fb56413f710 legacy=0x7fb5641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:26.173 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.171+0000 7fb568a63640 1 -- 172.21.15.156:0/2226531033 >> v1:172.21.15.156:6789/0 conn(0x7fb5640a98f0 legacy=0x7fb5640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:26.173 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.171+0000 7fb568a63640 10 monclient: _finish_hunting 0 2024-03-25T18:17:26.173 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.171+0000 7fb568a63640 1 monclient: found mon.noname-c 2024-03-25T18:17:26.173 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.171+0000 7fb568a63640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:26.173 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.171+0000 7fb568a63640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:17:26.173 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.171+0000 7fb568a63640 1 -- 172.21.15.156:0/2226531033 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5640b44c0 con 0x7fb5641430c0 2024-03-25T18:17:26.173 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.171+0000 7fb568a63640 10 monclient: _finish_auth 0 2024-03-25T18:17:26.173 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.171+0000 7fb568a63640 10 monclient: _check_auth_tickets 2024-03-25T18:17:26.173 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.171+0000 7fb568a63640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:26.173 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.171+0000 7fb568a63640 1 -- 172.21.15.156:0/2226531033 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fb55c001a10 con 0x7fb5641430c0 2024-03-25T18:17:26.173 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.171+0000 7fb56bcc9640 5 monclient: authenticate success, global_id 4636 2024-03-25T18:17:26.173 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.171+0000 7fb56bcc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:26.173 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.171+0000 7fb568a63640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:26.173 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.171+0000 7fb568a63640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:26.173 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.171+0000 7fb568a63640 10 monclient: dump: 2024-03-25T18:17:26.174 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:26.174 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:26.174 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:26.174 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:26.174 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:26.174 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:26.174 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:26.174 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:26.174 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:26.174 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:26.174 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.171+0000 7fb56bcc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:26.187 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.186+0000 7fb568a63640 1 -- 172.21.15.156:0/2226531033 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb55c001b70 con 0x7fb5641430c0 2024-03-25T18:17:26.187 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.186+0000 7fb568a63640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:26.187 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.186+0000 7fb568a63640 1 -- 172.21.15.156:0/2226531033 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fb55c001f80 con 0x7fb5641430c0 2024-03-25T18:17:26.187 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.186+0000 7fb56bcc9640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:26.187 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.186+0000 7fb56bcc9640 4 set_mon_vals no callback set 2024-03-25T18:17:26.187 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.186+0000 7fb568a63640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:26.187 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.186+0000 7fb568a63640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:26.187 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.186+0000 7fb568a63640 10 monclient: dump: 2024-03-25T18:17:26.187 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:26.187 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:26.187 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:26.188 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:26.188 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:26.188 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:26.188 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:26.188 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:26.188 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:26.188 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:26.189 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.186+0000 7fb56bcc9640 10 monclient: shutdown 2024-03-25T18:17:26.189 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.186+0000 7fb56bcc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:26.189 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.186+0000 7fb56bcc9640 1 -- 172.21.15.156:0/2226531033 >> v1:172.21.15.194:6789/0 conn(0x7fb5641430c0 legacy=0x7fb56414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:26.189 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.186+0000 7fb56bcc9640 1 -- 172.21.15.156:0/2226531033 shutdown_connections 2024-03-25T18:17:26.189 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.186+0000 7fb56bcc9640 1 -- 172.21.15.156:0/2226531033 >> 172.21.15.156:0/2226531033 conn(0x7fb5640b7ea0 msgr2=0x7fb5640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:26.189 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.187+0000 7fb56bcc9640 1 -- 172.21.15.156:0/2226531033 shutdown_connections 2024-03-25T18:17:26.189 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.187+0000 7fb56bcc9640 1 -- 172.21.15.156:0/2226531033 wait complete. 2024-03-25T18:17:26.190 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.187+0000 7fb56bcc9640 10 monclient: build_initial_monmap 2024-03-25T18:17:26.190 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.187+0000 7fb56bcc9640 10 monclient: monmap: 2024-03-25T18:17:26.190 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:17:26.190 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:26.190 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:17:26.190 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:17:26.190 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:26.190 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:17:26.190 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:17:26.190 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:17:26.190 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:17:26.190 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:26.190 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.188+0000 7fb56bcc9640 1 librados: starting msgr at 2024-03-25T18:17:26.190 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.188+0000 7fb56bcc9640 1 librados: starting objecter 2024-03-25T18:17:26.190 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.188+0000 7fb56bcc9640 1 Processor -- start 2024-03-25T18:17:26.190 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.188+0000 7fb56bcc9640 1 -- start start 2024-03-25T18:17:26.191 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.188+0000 7fb56bcc9640 1 librados: setting wanted keys 2024-03-25T18:17:26.191 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.188+0000 7fb56bcc9640 1 librados: calling monclient init 2024-03-25T18:17:26.191 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.188+0000 7fb56bcc9640 10 monclient: init 2024-03-25T18:17:26.192 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.189+0000 7fb56bcc9640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:26.192 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.189+0000 7fb56bcc9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:26.192 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.189+0000 7fb56bcc9640 10 monclient(hunting): picked mon.noname-a con 0x7fb5640a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:17:26.192 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.189+0000 7fb56bcc9640 10 monclient(hunting): picked mon.noname-c con 0x7fb56413f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:17:26.192 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.189+0000 7fb56bcc9640 10 monclient(hunting): picked mon.noname-b con 0x7fb5641430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:17:26.192 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.189+0000 7fb56bcc9640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb564137180 con 0x7fb5640a98f0 2024-03-25T18:17:26.192 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.189+0000 7fb56bcc9640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5640b74b0 con 0x7fb5641430c0 2024-03-25T18:17:26.192 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.189+0000 7fb56bcc9640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb564137d20 con 0x7fb56413f710 2024-03-25T18:17:26.192 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.189+0000 7fb56bcc9640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:26.192 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.189+0000 7fb56bcc9640 10 monclient(hunting): authenticate will time out at 7527.500021s 2024-03-25T18:17:26.193 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.192+0000 7fb569264640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7fb56413f710 0x7fb564137610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:48222/0 (socket says 172.21.15.156:48222) 2024-03-25T18:17:26.194 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.193+0000 7fb569264640 1 -- 172.21.15.156:0/2171432256 learned_addr learned my addr 172.21.15.156:0/2171432256 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:17:26.194 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.193+0000 7fb55a7fc640 10 client.?.objecter ms_handle_connect 0x7fb56413f710 2024-03-25T18:17:26.194 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.193+0000 7fb55a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:26.201 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.200+0000 7fb55a7fc640 1 -- 172.21.15.156:0/2171432256 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3581573973 0 0) 0x7fb564137d20 con 0x7fb56413f710 2024-03-25T18:17:26.202 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.200+0000 7fb55a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:26.202 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.200+0000 7fb55a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:26.202 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.200+0000 7fb55a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:26.202 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.200+0000 7fb55a7fc640 10 monclient(hunting): my global_id is 4643 2024-03-25T18:17:26.202 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.200+0000 7fb55a7fc640 1 -- 172.21.15.156:0/2171432256 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb548004eb0 con 0x7fb56413f710 2024-03-25T18:17:26.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.203+0000 7fb55a7fc640 10 client.?.objecter ms_handle_connect 0x7fb5641430c0 2024-03-25T18:17:26.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.203+0000 7fb55a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:26.220 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.219+0000 7fb55a7fc640 10 client.?.objecter ms_handle_connect 0x7fb5640a98f0 2024-03-25T18:17:26.220 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.219+0000 7fb55a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:26.222 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.221+0000 7fb55a7fc640 1 -- 172.21.15.156:0/2171432256 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1545537299 0 0) 0x7fb5640b74b0 con 0x7fb5641430c0 2024-03-25T18:17:26.222 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.221+0000 7fb55a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:26.222 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.221+0000 7fb55a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:26.222 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.221+0000 7fb55a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:26.222 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.221+0000 7fb55a7fc640 10 monclient(hunting): my global_id is 4639 2024-03-25T18:17:26.222 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.221+0000 7fb55a7fc640 1 -- 172.21.15.156:0/2171432256 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb564137d20 con 0x7fb5641430c0 2024-03-25T18:17:26.233 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.232+0000 7fb55a7fc640 1 -- 172.21.15.156:0/2171432256 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888152605 0 0) 0x7fb564137180 con 0x7fb5640a98f0 2024-03-25T18:17:26.233 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.232+0000 7fb55a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:26.233 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.232+0000 7fb55a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:26.234 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.232+0000 7fb55a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:26.234 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.232+0000 7fb55a7fc640 10 monclient(hunting): my global_id is 4635 2024-03-25T18:17:26.234 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.232+0000 7fb55a7fc640 1 -- 172.21.15.156:0/2171432256 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5640b74b0 con 0x7fb5640a98f0 2024-03-25T18:17:26.239 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.237+0000 7fb55a7fc640 1 -- 172.21.15.156:0/2171432256 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3690070938 0 0) 0x7fb564137d20 con 0x7fb5641430c0 2024-03-25T18:17:26.239 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.237+0000 7fb55a7fc640 1 -- 172.21.15.156:0/2171432256 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb564137180 con 0x7fb5641430c0 2024-03-25T18:17:26.239 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.237+0000 7fb55a7fc640 1 -- 172.21.15.156:0/2171432256 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fb55c001e80 con 0x7fb5641430c0 2024-03-25T18:17:26.239 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.237+0000 7fb55a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:26.239 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.237+0000 7fb55a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:26.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.237+0000 7fb55a7fc640 10 monclient(hunting): dump: 2024-03-25T18:17:26.240 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:26.240 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:26.240 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:26.240 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:26.240 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:26.240 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:26.240 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:26.240 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:26.240 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:26.240 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:26.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.237+0000 7fb55a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:26.241 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.238+0000 7fb55a7fc640 1 -- 172.21.15.156:0/2171432256 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 242385389 0 0) 0x7fb548004eb0 con 0x7fb56413f710 2024-03-25T18:17:26.241 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.238+0000 7fb55a7fc640 1 -- 172.21.15.156:0/2171432256 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb564137d20 con 0x7fb56413f710 2024-03-25T18:17:26.241 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.238+0000 7fb55a7fc640 1 -- 172.21.15.156:0/2171432256 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fb5540019d0 con 0x7fb56413f710 2024-03-25T18:17:26.241 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.238+0000 7fb55a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:26.241 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.238+0000 7fb55a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:26.241 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.238+0000 7fb55a7fc640 10 monclient(hunting): dump: 2024-03-25T18:17:26.241 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:26.241 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:26.241 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:26.241 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:26.241 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:26.241 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:26.242 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:26.242 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:26.242 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:26.242 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:26.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.244+0000 7fb55a7fc640 1 -- 172.21.15.156:0/2171432256 <== mon.2 v1:172.21.15.156:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 350386729 0 0) 0x7fb564137d20 con 0x7fb56413f710 2024-03-25T18:17:26.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.244+0000 7fb55a7fc640 1 -- 172.21.15.156:0/2171432256 >> v1:172.21.15.194:6789/0 conn(0x7fb5641430c0 legacy=0x7fb5641e2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:26.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.244+0000 7fb55a7fc640 1 -- 172.21.15.156:0/2171432256 >> v1:172.21.15.156:6789/0 conn(0x7fb5640a98f0 legacy=0x7fb5640b6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:26.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.244+0000 7fb55a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:17:26.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.244+0000 7fb55a7fc640 1 monclient: found mon.c 2024-03-25T18:17:26.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.244+0000 7fb55a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:26.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.244+0000 7fb55a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:17:26.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.244+0000 7fb55a7fc640 1 -- 172.21.15.156:0/2171432256 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb564138c40 con 0x7fb56413f710 2024-03-25T18:17:26.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.244+0000 7fb55a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:17:26.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.244+0000 7fb55a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:17:26.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.244+0000 7fb55a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:26.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.244+0000 7fb56bcc9640 5 monclient: authenticate success, global_id 4643 2024-03-25T18:17:26.247 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.244+0000 7fb56bcc9640 10 monclient: _renew_subs 2024-03-25T18:17:26.247 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.244+0000 7fb56bcc9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:17:26.247 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.244+0000 7fb56bcc9640 1 -- 172.21.15.156:0/2171432256 --> v1:172.21.15.156:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5641396b0 con 0x7fb56413f710 2024-03-25T18:17:26.247 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.244+0000 7fb56bcc9640 10 client.4643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:26.247 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.244+0000 7fb56bcc9640 10 monclient: _renew_subs 2024-03-25T18:17:26.247 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.244+0000 7fb56bcc9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:17:26.247 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.244+0000 7fb56bcc9640 1 -- 172.21.15.156:0/2171432256 --> v1:172.21.15.156:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5641dabe0 con 0x7fb56413f710 2024-03-25T18:17:26.247 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.244+0000 7fb56bcc9640 1 librados: init done 2024-03-25T18:17:26.248 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.245+0000 7fb56bcc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:26.248 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.245+0000 7fb56bcc9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:26.248 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.245+0000 7fb56bcc9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:17:26.248 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.245+0000 7fb56bcc9640 1 -- 172.21.15.156:0/2171432256 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5640627b0 con 0x7fb56413f710 2024-03-25T18:17:26.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.564+0000 7fb55a7fc640 1 -- 172.21.15.156:0/2171432256 <== mon.2 v1:172.21.15.156:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb554003610 con 0x7fb56413f710 2024-03-25T18:17:26.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.564+0000 7fb55a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:26.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.564+0000 7fb55a7fc640 1 -- 172.21.15.156:0/2171432256 <== mon.2 v1:172.21.15.156:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fb554003960 con 0x7fb56413f710 2024-03-25T18:17:26.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.564+0000 7fb55a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:26.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.564+0000 7fb55a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:26.565 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.564+0000 7fb55a7fc640 10 monclient: dump: 2024-03-25T18:17:26.566 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:26.566 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:26.566 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:26.566 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:26.566 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:26.566 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:26.566 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:26.566 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:26.566 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:26.566 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:26.566 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.564+0000 7fb55bfff640 4 set_mon_vals no callback set 2024-03-25T18:17:26.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.565+0000 7fb55a7fc640 1 -- 172.21.15.156:0/2171432256 <== mon.2 v1:172.21.15.156:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7fb554003b80 con 0x7fb56413f710 2024-03-25T18:17:26.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.565+0000 7fb569a65640 10 monclient: get_auth_request con 0x7fb54806dfd0 auth_method 0 2024-03-25T18:17:26.567 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.566+0000 7fb55a7fc640 10 client.4643.objecter ms_handle_connect 0x7fb54806dfd0 2024-03-25T18:17:26.573 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.571+0000 7fb55a7fc640 1 -- 172.21.15.156:0/2171432256 <== mon.2 v1:172.21.15.156:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7fb554054770 con 0x7fb56413f710 2024-03-25T18:17:26.573 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.571+0000 7fb55a7fc640 10 client.4643.objecter ms_dispatch 0x7fb564000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:26.573 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.571+0000 7fb55a7fc640 3 client.4643.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:26.573 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.571+0000 7fb55a7fc640 3 client.4643.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:26.573 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.571+0000 7fb55a7fc640 20 client.4643.objecter dump_active .. 0 homeless 2024-03-25T18:17:26.573 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.571+0000 7fb55a7fc640 1 -- 172.21.15.156:0/2171432256 <== mon.2 v1:172.21.15.156:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb554050280 con 0x7fb56413f710 2024-03-25T18:17:26.573 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.571+0000 7fb55a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:26.573 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.571+0000 7fb55a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:26.816 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.815+0000 7fb56bcc9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:26.817 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.815+0000 7fb56bcc9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:26.817 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.815+0000 7fb56bcc9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:17:26.817 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.815+0000 7fb56bcc9640 1 -- 172.21.15.156:0/2171432256 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb56413e230 con 0x7fb56413f710 2024-03-25T18:17:26.892 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.891+0000 7fb55a7fc640 1 -- 172.21.15.156:0/2171432256 <== mon.2 v1:172.21.15.156:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7fb5540521b0 con 0x7fb56413f710 2024-03-25T18:17:26.892 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.891+0000 7fb55a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:26.892 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.891+0000 7fb55a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:26.893 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:17:26.893 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:26.896 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.894+0000 7fb56bcc9640 10 librados: watch_flush enter 2024-03-25T18:17:26.896 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.894+0000 7fb56bcc9640 10 librados: watch_flush exit 2024-03-25T18:17:26.896 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.894+0000 7fb56bcc9640 20 client.4643.objecter shutdown clearing up homeless session... 2024-03-25T18:17:26.896 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.894+0000 7fb56bcc9640 10 client.4643.objecter successfully canceled tick 2024-03-25T18:17:26.896 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.894+0000 7fb56bcc9640 1 -- 172.21.15.156:0/2171432256 >> v1:172.21.15.194:6800/2946427713 conn(0x7fb54806dfd0 legacy=0x7fb548070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:26.896 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.894+0000 7fb56bcc9640 10 monclient: shutdown 2024-03-25T18:17:26.897 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.894+0000 7fb56bcc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:26.897 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.894+0000 7fb56bcc9640 1 -- 172.21.15.156:0/2171432256 >> v1:172.21.15.156:6790/0 conn(0x7fb56413f710 legacy=0x7fb564137610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:26.897 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.894+0000 7fb56bcc9640 1 -- 172.21.15.156:0/2171432256 shutdown_connections 2024-03-25T18:17:26.897 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.894+0000 7fb56bcc9640 1 -- 172.21.15.156:0/2171432256 >> 172.21.15.156:0/2171432256 conn(0x7fb5640b7ea0 msgr2=0x7fb564074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:26.897 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.894+0000 7fb56bcc9640 1 -- 172.21.15.156:0/2171432256 shutdown_connections 2024-03-25T18:17:26.897 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.894+0000 7fb56bcc9640 1 -- 172.21.15.156:0/2171432256 wait complete. 2024-03-25T18:17:26.897 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:26.894+0000 7fb56bcc9640 1 librados: shutdown 2024-03-25T18:17:26.907 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:33.713 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:33.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.831+0000 7fc612772640 10 monclient: get_monmap_and_config 2024-03-25T18:17:33.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.831+0000 7fc612772640 10 monclient: build_initial_monmap 2024-03-25T18:17:33.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.831+0000 7fc612772640 10 monclient: monmap: 2024-03-25T18:17:33.832 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:17:33.832 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:33.832 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:17:33.832324+0000 2024-03-25T18:17:33.832 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:17:33.832324+0000 2024-03-25T18:17:33.832 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:33.833 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:33.833 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:17:33.833 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:17:33.833 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:17:33.833 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:33.834 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.832+0000 7fc612772640 1 Processor -- start 2024-03-25T18:17:33.834 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.832+0000 7fc612772640 1 -- start start 2024-03-25T18:17:33.834 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.832+0000 7fc612772640 10 monclient: init 2024-03-25T18:17:33.834 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.832+0000 7fc612772640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:33.834 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.832+0000 7fc612772640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:33.834 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.832+0000 7fc612772640 10 monclient(hunting): picked mon.noname-a con 0x7fc60c0a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:17:33.834 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.832+0000 7fc612772640 10 monclient(hunting): picked mon.noname-b con 0x7fc60c13f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:17:33.834 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.832+0000 7fc612772640 10 monclient(hunting): picked mon.noname-c con 0x7fc60c1430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:17:33.834 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.832+0000 7fc612772640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc60c142520 con 0x7fc60c0a98f0 2024-03-25T18:17:33.834 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.832+0000 7fc612772640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc60c0b28a0 con 0x7fc60c1430c0 2024-03-25T18:17:33.834 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.833+0000 7fc612772640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc60c0b2a30 con 0x7fc60c13f710 2024-03-25T18:17:33.834 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.833+0000 7fc612772640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:33.834 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.833+0000 7fc612772640 10 monclient(hunting): authenticate will time out at 7535.143804s 2024-03-25T18:17:33.843 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.842+0000 7fc610d0f640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fc60c1430c0 0x7fc60c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:55064/0 (socket says 172.21.15.156:55064) 2024-03-25T18:17:33.843 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.842+0000 7fc610d0f640 1 -- 172.21.15.156:0/1265097516 learned_addr learned my addr 172.21.15.156:0/1265097516 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:17:33.853 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.852+0000 7fc60affd640 1 -- 172.21.15.156:0/1265097516 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3648537726 0 0) 0x7fc60c0b28a0 con 0x7fc60c1430c0 2024-03-25T18:17:33.854 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.852+0000 7fc60affd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:33.854 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.852+0000 7fc60affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:33.854 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.852+0000 7fc60affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:33.854 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.852+0000 7fc60affd640 10 monclient(hunting): my global_id is 4642 2024-03-25T18:17:33.854 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.852+0000 7fc60affd640 1 -- 172.21.15.156:0/1265097516 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5f4003520 con 0x7fc60c1430c0 2024-03-25T18:17:33.862 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.861+0000 7fc60affd640 1 -- 172.21.15.156:0/1265097516 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2554989607 0 0) 0x7fc5f4003520 con 0x7fc60c1430c0 2024-03-25T18:17:33.862 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.861+0000 7fc60affd640 1 -- 172.21.15.156:0/1265097516 >> v1:172.21.15.156:6790/0 conn(0x7fc60c13f710 legacy=0x7fc60c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:33.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.861+0000 7fc60affd640 1 -- 172.21.15.156:0/1265097516 >> v1:172.21.15.156:6789/0 conn(0x7fc60c0a98f0 legacy=0x7fc60c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:33.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.861+0000 7fc60affd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:33.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.861+0000 7fc60affd640 1 monclient: found mon.noname-c 2024-03-25T18:17:33.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.861+0000 7fc60affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:33.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.861+0000 7fc60affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:17:33.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.861+0000 7fc60affd640 1 -- 172.21.15.156:0/1265097516 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc60c0b44c0 con 0x7fc60c1430c0 2024-03-25T18:17:33.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.861+0000 7fc60affd640 10 monclient: _finish_auth 0 2024-03-25T18:17:33.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.861+0000 7fc60affd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:33.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.861+0000 7fc60affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:33.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.861+0000 7fc60affd640 1 -- 172.21.15.156:0/1265097516 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc604001a10 con 0x7fc60c1430c0 2024-03-25T18:17:33.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.861+0000 7fc612772640 5 monclient: authenticate success, global_id 4642 2024-03-25T18:17:33.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.861+0000 7fc612772640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:33.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.861+0000 7fc60affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:33.863 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.861+0000 7fc60affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:33.864 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.861+0000 7fc60affd640 10 monclient: dump: 2024-03-25T18:17:33.864 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:33.864 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:33.864 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:33.864 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:33.864 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:33.864 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:33.864 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:33.864 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:33.864 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:33.864 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:33.864 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.861+0000 7fc612772640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:33.890 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.889+0000 7fc60affd640 1 -- 172.21.15.156:0/1265097516 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc604001b70 con 0x7fc60c1430c0 2024-03-25T18:17:33.890 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.889+0000 7fc60affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:33.890 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.889+0000 7fc60affd640 1 -- 172.21.15.156:0/1265097516 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc604001f80 con 0x7fc60c1430c0 2024-03-25T18:17:33.890 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.889+0000 7fc612772640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:33.890 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.889+0000 7fc612772640 4 set_mon_vals no callback set 2024-03-25T18:17:33.890 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.889+0000 7fc60affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:33.890 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.889+0000 7fc60affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:33.890 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.889+0000 7fc60affd640 10 monclient: dump: 2024-03-25T18:17:33.891 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:33.891 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:33.891 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:33.891 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:33.891 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:33.891 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:33.891 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:33.891 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:33.891 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:33.891 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:33.892 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.890+0000 7fc612772640 10 monclient: shutdown 2024-03-25T18:17:33.892 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.890+0000 7fc612772640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:33.892 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.890+0000 7fc612772640 1 -- 172.21.15.156:0/1265097516 >> v1:172.21.15.194:6789/0 conn(0x7fc60c1430c0 legacy=0x7fc60c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:33.893 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.890+0000 7fc612772640 1 -- 172.21.15.156:0/1265097516 shutdown_connections 2024-03-25T18:17:33.893 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.890+0000 7fc612772640 1 -- 172.21.15.156:0/1265097516 >> 172.21.15.156:0/1265097516 conn(0x7fc60c0b7ea0 msgr2=0x7fc60c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:33.893 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.890+0000 7fc612772640 1 -- 172.21.15.156:0/1265097516 shutdown_connections 2024-03-25T18:17:33.893 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.890+0000 7fc612772640 1 -- 172.21.15.156:0/1265097516 wait complete. 2024-03-25T18:17:33.893 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.891+0000 7fc612772640 10 monclient: build_initial_monmap 2024-03-25T18:17:33.893 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.891+0000 7fc612772640 10 monclient: monmap: 2024-03-25T18:17:33.893 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:17:33.893 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:33.893 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:17:33.893 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:17:33.893 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:33.893 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:17:33.893 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:17:33.893 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:17:33.893 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:17:33.894 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:33.894 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.891+0000 7fc612772640 1 librados: starting msgr at 2024-03-25T18:17:33.894 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.891+0000 7fc612772640 1 librados: starting objecter 2024-03-25T18:17:33.894 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.892+0000 7fc612772640 1 Processor -- start 2024-03-25T18:17:33.895 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.892+0000 7fc612772640 1 -- start start 2024-03-25T18:17:33.895 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.892+0000 7fc612772640 1 librados: setting wanted keys 2024-03-25T18:17:33.895 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.892+0000 7fc612772640 1 librados: calling monclient init 2024-03-25T18:17:33.895 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.892+0000 7fc612772640 10 monclient: init 2024-03-25T18:17:33.895 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.892+0000 7fc612772640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:33.895 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.892+0000 7fc612772640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:33.895 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.892+0000 7fc612772640 10 monclient(hunting): picked mon.noname-c con 0x7fc60c0a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:17:33.895 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.892+0000 7fc612772640 10 monclient(hunting): picked mon.noname-b con 0x7fc60c13f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:17:33.895 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.893+0000 7fc612772640 10 monclient(hunting): picked mon.noname-a con 0x7fc60c1430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:17:33.895 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.893+0000 7fc612772640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc60c13cc30 con 0x7fc60c1430c0 2024-03-25T18:17:33.895 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.893+0000 7fc612772640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc60c0b71e0 con 0x7fc60c13f710 2024-03-25T18:17:33.895 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.893+0000 7fc612772640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc60c13afc0 con 0x7fc60c0a98f0 2024-03-25T18:17:33.896 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.893+0000 7fc612772640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:33.896 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.893+0000 7fc612772640 10 monclient(hunting): authenticate will time out at 7535.203857s 2024-03-25T18:17:33.903 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.902+0000 7fc610d0f640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fc60c1430c0 0x7fc60c1e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:45600/0 (socket says 172.21.15.156:45600) 2024-03-25T18:17:33.903 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.902+0000 7fc610d0f640 1 -- 172.21.15.156:0/3595114325 learned_addr learned my addr 172.21.15.156:0/3595114325 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:17:33.905 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.904+0000 7fc608ff9640 10 client.?.objecter ms_handle_connect 0x7fc60c1430c0 2024-03-25T18:17:33.905 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.904+0000 7fc608ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:33.915 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.915+0000 7fc608ff9640 10 client.?.objecter ms_handle_connect 0x7fc60c0a98f0 2024-03-25T18:17:33.916 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.915+0000 7fc608ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:33.919 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.918+0000 7fc608ff9640 10 client.?.objecter ms_handle_connect 0x7fc60c13f710 2024-03-25T18:17:33.919 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.918+0000 7fc608ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:33.925 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.924+0000 7fc608ff9640 1 -- 172.21.15.156:0/3595114325 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760479606 0 0) 0x7fc60c13cc30 con 0x7fc60c1430c0 2024-03-25T18:17:33.925 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.924+0000 7fc608ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:33.925 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.924+0000 7fc608ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:33.925 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.924+0000 7fc608ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:33.925 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.924+0000 7fc608ff9640 10 monclient(hunting): my global_id is 4641 2024-03-25T18:17:33.926 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.924+0000 7fc608ff9640 1 -- 172.21.15.156:0/3595114325 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5f8004eb0 con 0x7fc60c1430c0 2024-03-25T18:17:33.935 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.933+0000 7fc608ff9640 1 -- 172.21.15.156:0/3595114325 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 831791493 0 0) 0x7fc60c0b71e0 con 0x7fc60c13f710 2024-03-25T18:17:33.935 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.933+0000 7fc608ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:33.935 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.933+0000 7fc608ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:33.935 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.933+0000 7fc608ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:33.935 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.933+0000 7fc608ff9640 10 monclient(hunting): my global_id is 4645 2024-03-25T18:17:33.935 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.933+0000 7fc608ff9640 1 -- 172.21.15.156:0/3595114325 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc60c13cc30 con 0x7fc60c13f710 2024-03-25T18:17:33.935 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.934+0000 7fc608ff9640 1 -- 172.21.15.156:0/3595114325 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610782674 0 0) 0x7fc60c13afc0 con 0x7fc60c0a98f0 2024-03-25T18:17:33.936 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.934+0000 7fc608ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:33.936 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.934+0000 7fc608ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:33.936 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.934+0000 7fc608ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:33.936 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.934+0000 7fc608ff9640 10 monclient(hunting): my global_id is 4649 2024-03-25T18:17:33.936 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.934+0000 7fc608ff9640 1 -- 172.21.15.156:0/3595114325 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc60c0b71e0 con 0x7fc60c0a98f0 2024-03-25T18:17:33.937 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.936+0000 7fc608ff9640 1 -- 172.21.15.156:0/3595114325 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2952310240 0 0) 0x7fc5f8004eb0 con 0x7fc60c1430c0 2024-03-25T18:17:33.937 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.936+0000 7fc608ff9640 1 -- 172.21.15.156:0/3595114325 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc60c13afc0 con 0x7fc60c1430c0 2024-03-25T18:17:33.937 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.936+0000 7fc608ff9640 1 -- 172.21.15.156:0/3595114325 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc604001ec0 con 0x7fc60c1430c0 2024-03-25T18:17:33.937 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.936+0000 7fc608ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:33.937 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.936+0000 7fc608ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:33.937 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.936+0000 7fc608ff9640 10 monclient(hunting): dump: 2024-03-25T18:17:33.937 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:33.937 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:33.937 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:33.937 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:33.937 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:33.937 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:33.938 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:33.938 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:33.938 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:33.938 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:33.938 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.936+0000 7fc608ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:33.945 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.943+0000 7fc608ff9640 1 -- 172.21.15.156:0/3595114325 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4141112641 0 0) 0x7fc60c13cc30 con 0x7fc60c13f710 2024-03-25T18:17:33.945 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.943+0000 7fc608ff9640 1 -- 172.21.15.156:0/3595114325 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5f8004eb0 con 0x7fc60c13f710 2024-03-25T18:17:33.945 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.943+0000 7fc608ff9640 1 -- 172.21.15.156:0/3595114325 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc6000018b0 con 0x7fc60c13f710 2024-03-25T18:17:33.945 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.943+0000 7fc608ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:33.945 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.943+0000 7fc608ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:33.945 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.943+0000 7fc608ff9640 10 monclient(hunting): dump: 2024-03-25T18:17:33.945 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:33.945 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:33.945 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:33.945 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:33.945 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:33.945 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:33.945 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:33.945 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:33.945 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:33.945 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:33.947 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.945+0000 7fc608ff9640 1 -- 172.21.15.156:0/3595114325 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 581422975 0 0) 0x7fc60c0b71e0 con 0x7fc60c0a98f0 2024-03-25T18:17:33.947 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.945+0000 7fc608ff9640 1 -- 172.21.15.156:0/3595114325 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc60c13cc30 con 0x7fc60c0a98f0 2024-03-25T18:17:33.947 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.945+0000 7fc608ff9640 1 -- 172.21.15.156:0/3595114325 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc5fc001830 con 0x7fc60c0a98f0 2024-03-25T18:17:33.947 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.945+0000 7fc608ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:33.947 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.945+0000 7fc608ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:33.947 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.945+0000 7fc608ff9640 10 monclient(hunting): dump: 2024-03-25T18:17:33.947 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:33.947 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:33.947 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:33.947 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:33.947 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:33.947 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:33.947 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:33.948 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:33.948 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:33.948 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:33.956 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.955+0000 7fc608ff9640 1 -- 172.21.15.156:0/3595114325 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2271611521 0 0) 0x7fc5f8004eb0 con 0x7fc60c13f710 2024-03-25T18:17:33.956 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.955+0000 7fc608ff9640 1 -- 172.21.15.156:0/3595114325 >> v1:172.21.15.156:6790/0 conn(0x7fc60c0a98f0 legacy=0x7fc60c13b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:33.956 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.955+0000 7fc608ff9640 1 -- 172.21.15.156:0/3595114325 >> v1:172.21.15.156:6789/0 conn(0x7fc60c1430c0 legacy=0x7fc60c1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:33.956 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.955+0000 7fc608ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:17:33.956 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.955+0000 7fc608ff9640 1 monclient: found mon.b 2024-03-25T18:17:33.956 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.955+0000 7fc608ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:33.956 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.955+0000 7fc608ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:17:33.957 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.955+0000 7fc608ff9640 1 -- 172.21.15.156:0/3595114325 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc60c13b2a0 con 0x7fc60c13f710 2024-03-25T18:17:33.957 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.955+0000 7fc608ff9640 10 monclient: _finish_auth 0 2024-03-25T18:17:33.957 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.955+0000 7fc608ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:17:33.957 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.955+0000 7fc608ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:33.957 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.955+0000 7fc612772640 5 monclient: authenticate success, global_id 4645 2024-03-25T18:17:33.957 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.955+0000 7fc612772640 10 monclient: _renew_subs 2024-03-25T18:17:33.957 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.955+0000 7fc612772640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:17:33.957 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.955+0000 7fc612772640 1 -- 172.21.15.156:0/3595114325 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc60c13b580 con 0x7fc60c13f710 2024-03-25T18:17:33.957 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.955+0000 7fc612772640 10 client.4645.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:33.957 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.955+0000 7fc612772640 10 monclient: _renew_subs 2024-03-25T18:17:33.957 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.955+0000 7fc612772640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:17:33.957 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.955+0000 7fc612772640 1 -- 172.21.15.156:0/3595114325 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc60c0b7ca0 con 0x7fc60c13f710 2024-03-25T18:17:33.957 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.955+0000 7fc612772640 1 librados: init done 2024-03-25T18:17:33.958 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.956+0000 7fc612772640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:33.958 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.956+0000 7fc612772640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:33.958 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.956+0000 7fc612772640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:17:33.958 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:33.956+0000 7fc612772640 1 -- 172.21.15.156:0/3595114325 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc60c0627b0 con 0x7fc60c13f710 2024-03-25T18:17:34.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.244+0000 7fc608ff9640 1 -- 172.21.15.156:0/3595114325 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6000035f0 con 0x7fc60c13f710 2024-03-25T18:17:34.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.244+0000 7fc608ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:34.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.244+0000 7fc608ff9640 1 -- 172.21.15.156:0/3595114325 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc600003a40 con 0x7fc60c13f710 2024-03-25T18:17:34.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.244+0000 7fc608ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:34.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.244+0000 7fc608ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:34.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.244+0000 7fc608ff9640 10 monclient: dump: 2024-03-25T18:17:34.245 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:34.245 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:34.245 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:34.246 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:34.246 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:34.246 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:34.246 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:34.246 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:34.246 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:34.246 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:34.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.244+0000 7fc60a7fc640 4 set_mon_vals no callback set 2024-03-25T18:17:34.247 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.244+0000 7fc608ff9640 1 -- 172.21.15.156:0/3595114325 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7fc600003c60 con 0x7fc60c13f710 2024-03-25T18:17:34.247 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.245+0000 7fc60bfff640 10 monclient: get_auth_request con 0x7fc5f806e160 auth_method 0 2024-03-25T18:17:34.247 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.245+0000 7fc608ff9640 10 client.4645.objecter ms_handle_connect 0x7fc5f806e160 2024-03-25T18:17:34.255 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.253+0000 7fc608ff9640 1 -- 172.21.15.156:0/3595114325 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7fc600053570 con 0x7fc60c13f710 2024-03-25T18:17:34.255 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.253+0000 7fc608ff9640 10 client.4645.objecter ms_dispatch 0x7fc60c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:34.255 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.253+0000 7fc608ff9640 3 client.4645.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:34.255 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.253+0000 7fc608ff9640 3 client.4645.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:34.255 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.254+0000 7fc608ff9640 20 client.4645.objecter dump_active .. 0 homeless 2024-03-25T18:17:34.255 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.254+0000 7fc608ff9640 1 -- 172.21.15.156:0/3595114325 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc60004f080 con 0x7fc60c13f710 2024-03-25T18:17:34.255 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.254+0000 7fc608ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:34.255 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.254+0000 7fc608ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:34.479 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.478+0000 7fc612772640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:34.479 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.478+0000 7fc612772640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:34.479 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.478+0000 7fc612772640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:17:34.479 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.478+0000 7fc612772640 1 -- 172.21.15.156:0/3595114325 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc60c0b7390 con 0x7fc60c13f710 2024-03-25T18:17:34.585 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.584+0000 7fc608ff9640 1 -- 172.21.15.156:0/3595114325 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7fc60c0b7390 con 0x7fc60c13f710 2024-03-25T18:17:34.585 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.584+0000 7fc608ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:34.585 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.584+0000 7fc608ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:34.586 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:17:34.586 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:34.588 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.587+0000 7fc612772640 10 librados: watch_flush enter 2024-03-25T18:17:34.588 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.587+0000 7fc612772640 10 librados: watch_flush exit 2024-03-25T18:17:34.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.587+0000 7fc612772640 20 client.4645.objecter shutdown clearing up homeless session... 2024-03-25T18:17:34.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.587+0000 7fc612772640 10 client.4645.objecter successfully canceled tick 2024-03-25T18:17:34.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.587+0000 7fc612772640 1 -- 172.21.15.156:0/3595114325 >> v1:172.21.15.194:6800/2946427713 conn(0x7fc5f806e160 legacy=0x7fc5f8070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:34.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.587+0000 7fc612772640 10 monclient: shutdown 2024-03-25T18:17:34.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.587+0000 7fc612772640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:34.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.587+0000 7fc612772640 1 -- 172.21.15.156:0/3595114325 >> v1:172.21.15.194:6789/0 conn(0x7fc60c13f710 legacy=0x7fc60c139e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:34.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.587+0000 7fc612772640 1 -- 172.21.15.156:0/3595114325 shutdown_connections 2024-03-25T18:17:34.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.587+0000 7fc612772640 1 -- 172.21.15.156:0/3595114325 >> 172.21.15.156:0/3595114325 conn(0x7fc60c0b7ea0 msgr2=0x7fc60c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:34.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.587+0000 7fc612772640 1 -- 172.21.15.156:0/3595114325 shutdown_connections 2024-03-25T18:17:34.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.587+0000 7fc612772640 1 -- 172.21.15.156:0/3595114325 wait complete. 2024-03-25T18:17:34.589 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:34.587+0000 7fc612772640 1 librados: shutdown 2024-03-25T18:17:34.600 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:39.936 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:09:45.926 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:39.990 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:09:45.981 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:40.023 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:09:46.013 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:40.039 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:09:46.006 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:40.044 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:09:46.011 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:40.044 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:09:46.034 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:40.057 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:09:46.047 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:40.080 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:09:46.047 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:40.097 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:09:46.087 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:41.405 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:41.524 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.523+0000 7f4ac7127640 10 monclient: get_monmap_and_config 2024-03-25T18:17:41.524 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.523+0000 7f4ac7127640 10 monclient: build_initial_monmap 2024-03-25T18:17:41.524 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.523+0000 7f4ac7127640 10 monclient: monmap: 2024-03-25T18:17:41.524 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:17:41.524 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:41.524 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:17:41.524539+0000 2024-03-25T18:17:41.525 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:17:41.524539+0000 2024-03-25T18:17:41.525 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:41.525 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:41.525 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:17:41.525 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:17:41.525 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:17:41.525 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:41.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.524+0000 7f4ac7127640 1 Processor -- start 2024-03-25T18:17:41.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.524+0000 7f4ac7127640 1 -- start start 2024-03-25T18:17:41.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.524+0000 7f4ac7127640 10 monclient: init 2024-03-25T18:17:41.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.525+0000 7f4ac7127640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:41.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.525+0000 7f4ac7127640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:17:41.526 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.525+0000 7f4ac7127640 10 monclient(hunting): picked mon.noname-b con 0x7f4ac00b4d60 addr v1:172.21.15.156:6790/0 2024-03-25T18:17:41.527 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.525+0000 7f4ac7127640 10 monclient(hunting): picked mon.noname-a con 0x7f4ac00b1730 addr v1:172.21.15.156:6789/0 2024-03-25T18:17:41.527 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.525+0000 7f4ac7127640 10 monclient(hunting): picked mon.noname-c con 0x7f4ac014be90 addr v1:172.21.15.194:6789/0 2024-03-25T18:17:41.527 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.525+0000 7f4ac7127640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac00af940 con 0x7f4ac00b1730 2024-03-25T18:17:41.527 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.525+0000 7f4ac7127640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac00b0a70 con 0x7f4ac014be90 2024-03-25T18:17:41.527 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.525+0000 7f4ac7127640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac00b0d50 con 0x7f4ac00b4d60 2024-03-25T18:17:41.527 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.525+0000 7f4ac7127640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:41.527 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.525+0000 7f4ac7127640 10 monclient(hunting): authenticate will time out at 7542.835904s 2024-03-25T18:17:41.533 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.532+0000 7f4ac56c4640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f4ac014be90 0x7f4ac014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:37788/0 (socket says 172.21.15.156:37788) 2024-03-25T18:17:41.533 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.532+0000 7f4ac56c4640 1 -- 172.21.15.156:0/20620649 learned_addr learned my addr 172.21.15.156:0/20620649 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:17:41.545 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.545+0000 7f4abf7fe640 1 -- 172.21.15.156:0/20620649 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658889910 0 0) 0x7f4ac00b0a70 con 0x7f4ac014be90 2024-03-25T18:17:41.546 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.545+0000 7f4abf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:41.546 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.545+0000 7f4abf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:41.546 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.545+0000 7f4abf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:41.546 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.545+0000 7f4abf7fe640 10 monclient(hunting): my global_id is 4648 2024-03-25T18:17:41.546 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.545+0000 7f4abf7fe640 1 -- 172.21.15.156:0/20620649 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ab0003520 con 0x7f4ac014be90 2024-03-25T18:17:41.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.546+0000 7f4abf7fe640 1 -- 172.21.15.156:0/20620649 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528394240 0 0) 0x7f4ac00b0d50 con 0x7f4ac00b4d60 2024-03-25T18:17:41.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.546+0000 7f4abf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:41.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.546+0000 7f4abf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:41.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.546+0000 7f4abf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:41.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.546+0000 7f4abf7fe640 10 monclient(hunting): my global_id is 4652 2024-03-25T18:17:41.547 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.546+0000 7f4abf7fe640 1 -- 172.21.15.156:0/20620649 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ac00b0a70 con 0x7f4ac00b4d60 2024-03-25T18:17:41.568 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.567+0000 7f4abf7fe640 1 -- 172.21.15.156:0/20620649 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032329552 0 0) 0x7f4ac00af940 con 0x7f4ac00b1730 2024-03-25T18:17:41.568 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.567+0000 7f4abf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:41.568 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.567+0000 7f4abf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:41.568 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.567+0000 7f4abf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:41.568 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.567+0000 7f4abf7fe640 10 monclient(hunting): my global_id is 4644 2024-03-25T18:17:41.568 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.567+0000 7f4abf7fe640 1 -- 172.21.15.156:0/20620649 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ac00b0d50 con 0x7f4ac00b1730 2024-03-25T18:17:41.570 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.568+0000 7f4abf7fe640 1 -- 172.21.15.156:0/20620649 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1424311651 0 0) 0x7f4ab0003520 con 0x7f4ac014be90 2024-03-25T18:17:41.570 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.568+0000 7f4abf7fe640 1 -- 172.21.15.156:0/20620649 >> v1:172.21.15.156:6790/0 conn(0x7f4ac00b4d60 legacy=0x7f4ac013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:41.570 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.568+0000 7f4abf7fe640 1 -- 172.21.15.156:0/20620649 >> v1:172.21.15.156:6789/0 conn(0x7f4ac00b1730 legacy=0x7f4ac00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:41.570 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.568+0000 7f4abf7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:17:41.570 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.568+0000 7f4abf7fe640 1 monclient: found mon.noname-c 2024-03-25T18:17:41.570 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.568+0000 7f4abf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:41.570 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.568+0000 7f4abf7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:17:41.570 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.568+0000 7f4abf7fe640 1 -- 172.21.15.156:0/20620649 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac00b1030 con 0x7f4ac014be90 2024-03-25T18:17:41.570 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.568+0000 7f4abf7fe640 10 monclient: _finish_auth 0 2024-03-25T18:17:41.570 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.568+0000 7f4abf7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:17:41.570 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.568+0000 7f4abf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:41.570 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.568+0000 7f4abf7fe640 1 -- 172.21.15.156:0/20620649 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f4ab8001a10 con 0x7f4ac014be90 2024-03-25T18:17:41.570 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.568+0000 7f4abf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:41.571 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.568+0000 7f4abf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:41.571 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.568+0000 7f4abf7fe640 10 monclient: dump: 2024-03-25T18:17:41.571 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:41.571 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:41.571 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:41.571 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:41.571 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:41.571 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:41.571 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:41.571 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:41.571 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:41.571 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:41.571 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.568+0000 7f4ac7127640 5 monclient: authenticate success, global_id 4648 2024-03-25T18:17:41.571 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.568+0000 7f4ac7127640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:41.598 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.597+0000 7f4abf7fe640 1 -- 172.21.15.156:0/20620649 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ab8001b70 con 0x7f4ac014be90 2024-03-25T18:17:41.598 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.597+0000 7f4abf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:41.599 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.597+0000 7f4abf7fe640 1 -- 172.21.15.156:0/20620649 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f4ab8001f80 con 0x7f4ac014be90 2024-03-25T18:17:41.601 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.597+0000 7f4ac7127640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:41.601 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.597+0000 7f4ac7127640 4 set_mon_vals no callback set 2024-03-25T18:17:41.602 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.597+0000 7f4abf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:41.602 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.597+0000 7f4abf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:41.602 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.597+0000 7f4abf7fe640 10 monclient: dump: 2024-03-25T18:17:41.602 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:41.602 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:41.602 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:41.602 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:41.602 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:41.602 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:41.602 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:41.602 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:41.602 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:41.602 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:41.604 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.598+0000 7f4ac7127640 10 monclient: shutdown 2024-03-25T18:17:41.604 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.598+0000 7f4ac7127640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:41.604 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.598+0000 7f4ac7127640 1 -- 172.21.15.156:0/20620649 >> v1:172.21.15.194:6789/0 conn(0x7f4ac014be90 legacy=0x7f4ac014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:41.605 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.598+0000 7f4ac7127640 1 -- 172.21.15.156:0/20620649 shutdown_connections 2024-03-25T18:17:41.605 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.598+0000 7f4ac7127640 1 -- 172.21.15.156:0/20620649 >> 172.21.15.156:0/20620649 conn(0x7f4ac00617a0 msgr2=0x7f4ac0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:41.605 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.598+0000 7f4ac7127640 1 -- 172.21.15.156:0/20620649 shutdown_connections 2024-03-25T18:17:41.605 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.598+0000 7f4ac7127640 1 -- 172.21.15.156:0/20620649 wait complete. 2024-03-25T18:17:41.605 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.599+0000 7f4ac7127640 10 monclient: build_initial_monmap 2024-03-25T18:17:41.605 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.599+0000 7f4ac7127640 10 monclient: monmap: 2024-03-25T18:17:41.605 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:17:41.605 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:41.605 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:17:41.605 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:17:41.605 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:41.605 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:17:41.605 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:17:41.605 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:17:41.606 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:17:41.606 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:41.606 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.599+0000 7f4ac7127640 1 librados: starting msgr at 2024-03-25T18:17:41.606 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.599+0000 7f4ac7127640 1 librados: starting objecter 2024-03-25T18:17:41.606 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.600+0000 7f4ac7127640 1 Processor -- start 2024-03-25T18:17:41.606 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.600+0000 7f4ac7127640 1 -- start start 2024-03-25T18:17:41.606 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.600+0000 7f4ac7127640 1 librados: setting wanted keys 2024-03-25T18:17:41.606 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.600+0000 7f4ac7127640 1 librados: calling monclient init 2024-03-25T18:17:41.606 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.600+0000 7f4ac7127640 10 monclient: init 2024-03-25T18:17:41.606 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.601+0000 7f4ac7127640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:41.606 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.601+0000 7f4ac7127640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:17:41.606 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.601+0000 7f4ac7127640 10 monclient(hunting): picked mon.noname-a con 0x7f4ac00b1730 addr v1:172.21.15.156:6789/0 2024-03-25T18:17:41.606 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.601+0000 7f4ac7127640 10 monclient(hunting): picked mon.noname-b con 0x7f4ac00b4d60 addr v1:172.21.15.194:6789/0 2024-03-25T18:17:41.606 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.601+0000 7f4ac7127640 10 monclient(hunting): picked mon.noname-c con 0x7f4ac014be90 addr v1:172.21.15.156:6790/0 2024-03-25T18:17:41.606 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.601+0000 7f4ac7127640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac01391b0 con 0x7f4ac00b1730 2024-03-25T18:17:41.606 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.601+0000 7f4ac7127640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac0132750 con 0x7f4ac00b4d60 2024-03-25T18:17:41.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.601+0000 7f4ac7127640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac0138e40 con 0x7f4ac014be90 2024-03-25T18:17:41.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.601+0000 7f4ac7127640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:41.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.601+0000 7f4ac7127640 10 monclient(hunting): authenticate will time out at 7542.912062s 2024-03-25T18:17:41.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.608+0000 7f4ac56c4640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f4ac014be90 0x7f4ac01db2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:46928/0 (socket says 172.21.15.156:46928) 2024-03-25T18:17:41.609 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.608+0000 7f4ac56c4640 1 -- 172.21.15.156:0/291717484 learned_addr learned my addr 172.21.15.156:0/291717484 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:17:41.618 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.617+0000 7f4abd7fa640 10 client.?.objecter ms_handle_connect 0x7f4ac014be90 2024-03-25T18:17:41.618 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.617+0000 7f4abd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:41.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.631+0000 7f4abd7fa640 10 client.?.objecter ms_handle_connect 0x7f4ac00b1730 2024-03-25T18:17:41.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.631+0000 7f4abd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:41.639 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.638+0000 7f4abd7fa640 10 client.?.objecter ms_handle_connect 0x7f4ac00b4d60 2024-03-25T18:17:41.639 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.638+0000 7f4abd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:41.640 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.639+0000 7f4abd7fa640 1 -- 172.21.15.156:0/291717484 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3638391374 0 0) 0x7f4ac0138e40 con 0x7f4ac014be90 2024-03-25T18:17:41.640 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.639+0000 7f4abd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:17:41.640 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.639+0000 7f4abd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:41.640 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.639+0000 7f4abd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:41.640 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.639+0000 7f4abd7fa640 10 monclient(hunting): my global_id is 4655 2024-03-25T18:17:41.641 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.639+0000 7f4abd7fa640 1 -- 172.21.15.156:0/291717484 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aa4004eb0 con 0x7f4ac014be90 2024-03-25T18:17:41.643 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.642+0000 7f4abd7fa640 1 -- 172.21.15.156:0/291717484 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3800542034 0 0) 0x7f4ac01391b0 con 0x7f4ac00b1730 2024-03-25T18:17:41.643 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.642+0000 7f4abd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:17:41.643 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.642+0000 7f4abd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:41.643 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.642+0000 7f4abd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:41.643 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.642+0000 7f4abd7fa640 10 monclient(hunting): my global_id is 4647 2024-03-25T18:17:41.643 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.642+0000 7f4abd7fa640 1 -- 172.21.15.156:0/291717484 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ac0138e40 con 0x7f4ac00b1730 2024-03-25T18:17:41.646 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.645+0000 7f4abd7fa640 1 -- 172.21.15.156:0/291717484 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799643408 0 0) 0x7f4ac0132750 con 0x7f4ac00b4d60 2024-03-25T18:17:41.647 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.645+0000 7f4abd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:17:41.647 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.645+0000 7f4abd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:41.647 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.645+0000 7f4abd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:41.647 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.645+0000 7f4abd7fa640 10 monclient(hunting): my global_id is 4651 2024-03-25T18:17:41.647 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.645+0000 7f4abd7fa640 1 -- 172.21.15.156:0/291717484 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ac01391b0 con 0x7f4ac00b4d60 2024-03-25T18:17:41.661 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.660+0000 7f4abd7fa640 1 -- 172.21.15.156:0/291717484 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 552714970 0 0) 0x7f4ac01391b0 con 0x7f4ac00b4d60 2024-03-25T18:17:41.662 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.660+0000 7f4abd7fa640 1 -- 172.21.15.156:0/291717484 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ac0132750 con 0x7f4ac00b4d60 2024-03-25T18:17:41.662 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.660+0000 7f4abd7fa640 1 -- 172.21.15.156:0/291717484 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f4aac001830 con 0x7f4ac00b4d60 2024-03-25T18:17:41.662 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.660+0000 7f4abd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:41.662 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.660+0000 7f4abd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:41.662 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.660+0000 7f4abd7fa640 10 monclient(hunting): dump: 2024-03-25T18:17:41.662 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:41.662 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:41.662 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:41.662 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:41.662 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:41.662 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:41.662 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:41.662 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:41.662 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:41.663 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:41.663 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.660+0000 7f4abd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:41.666 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.665+0000 7f4abd7fa640 1 -- 172.21.15.156:0/291717484 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1562219119 0 0) 0x7f4ac0138e40 con 0x7f4ac00b1730 2024-03-25T18:17:41.666 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.665+0000 7f4abd7fa640 1 -- 172.21.15.156:0/291717484 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ac01391b0 con 0x7f4ac00b1730 2024-03-25T18:17:41.666 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.665+0000 7f4abd7fa640 1 -- 172.21.15.156:0/291717484 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f4ab4001970 con 0x7f4ac00b1730 2024-03-25T18:17:41.666 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.665+0000 7f4abd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:41.667 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.665+0000 7f4abd7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:41.667 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.665+0000 7f4abd7fa640 10 monclient(hunting): dump: 2024-03-25T18:17:41.667 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:41.667 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:41.667 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:41.667 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:41.667 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:41.667 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:41.667 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:41.667 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:41.667 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:41.667 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:41.668 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.666+0000 7f4abd7fa640 1 -- 172.21.15.156:0/291717484 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3893887067 0 0) 0x7f4aa4004eb0 con 0x7f4ac014be90 2024-03-25T18:17:41.668 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.666+0000 7f4abd7fa640 1 -- 172.21.15.156:0/291717484 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ac0138e40 con 0x7f4ac014be90 2024-03-25T18:17:41.668 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.666+0000 7f4abd7fa640 1 -- 172.21.15.156:0/291717484 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f4ab8001ec0 con 0x7f4ac014be90 2024-03-25T18:17:41.668 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.666+0000 7f4abd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:41.668 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.666+0000 7f4abd7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:41.668 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.666+0000 7f4abd7fa640 10 monclient(hunting): dump: 2024-03-25T18:17:41.668 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:41.668 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:41.668 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:41.668 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:41.668 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:41.668 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:41.669 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:41.669 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:41.669 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:41.669 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:41.673 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.672+0000 7f4abd7fa640 1 -- 172.21.15.156:0/291717484 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3482304958 0 0) 0x7f4ac0132750 con 0x7f4ac00b4d60 2024-03-25T18:17:41.673 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.672+0000 7f4abd7fa640 1 -- 172.21.15.156:0/291717484 >> v1:172.21.15.156:6790/0 conn(0x7f4ac014be90 legacy=0x7f4ac01db2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:41.673 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.672+0000 7f4abd7fa640 1 -- 172.21.15.156:0/291717484 >> v1:172.21.15.156:6789/0 conn(0x7f4ac00b1730 legacy=0x7f4ac0132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:41.673 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.672+0000 7f4abd7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:17:41.673 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.672+0000 7f4abd7fa640 1 monclient: found mon.b 2024-03-25T18:17:41.674 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.672+0000 7f4abd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:41.674 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.672+0000 7f4abd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:17:41.674 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.672+0000 7f4abd7fa640 1 -- 172.21.15.156:0/291717484 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac0137c00 con 0x7f4ac00b4d60 2024-03-25T18:17:41.674 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.672+0000 7f4abd7fa640 10 monclient: _finish_auth 0 2024-03-25T18:17:41.674 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.672+0000 7f4abd7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:17:41.674 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.672+0000 7f4abd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:41.674 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.672+0000 7f4ac7127640 5 monclient: authenticate success, global_id 4651 2024-03-25T18:17:41.674 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.672+0000 7f4ac7127640 10 monclient: _renew_subs 2024-03-25T18:17:41.674 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.672+0000 7f4ac7127640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:17:41.674 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.672+0000 7f4ac7127640 1 -- 172.21.15.156:0/291717484 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ac0138050 con 0x7f4ac00b4d60 2024-03-25T18:17:41.674 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.672+0000 7f4ac7127640 10 client.4651.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:41.674 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.672+0000 7f4ac7127640 10 monclient: _renew_subs 2024-03-25T18:17:41.674 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.672+0000 7f4ac7127640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:17:41.674 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.672+0000 7f4ac7127640 1 -- 172.21.15.156:0/291717484 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ac0145080 con 0x7f4ac00b4d60 2024-03-25T18:17:41.675 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.672+0000 7f4ac7127640 1 librados: init done 2024-03-25T18:17:41.675 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.673+0000 7f4ac7127640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:41.675 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.673+0000 7f4ac7127640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:41.675 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.673+0000 7f4ac7127640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:17:41.675 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.673+0000 7f4ac7127640 1 -- 172.21.15.156:0/291717484 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ac01401f0 con 0x7f4ac00b4d60 2024-03-25T18:17:41.958 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.957+0000 7f4abd7fa640 1 -- 172.21.15.156:0/291717484 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4aac001d40 con 0x7f4ac00b4d60 2024-03-25T18:17:41.958 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.957+0000 7f4abd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:41.958 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.957+0000 7f4abd7fa640 1 -- 172.21.15.156:0/291717484 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f4aac0037f0 con 0x7f4ac00b4d60 2024-03-25T18:17:41.959 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.957+0000 7f4abd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:41.959 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.957+0000 7f4abd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:41.959 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.957+0000 7f4abd7fa640 10 monclient: dump: 2024-03-25T18:17:41.959 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:41.959 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:41.959 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:41.959 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:41.959 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:41.959 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:41.959 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:41.959 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:41.959 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:41.959 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:41.959 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.957+0000 7f4abeffd640 4 set_mon_vals no callback set 2024-03-25T18:17:41.960 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.958+0000 7f4abd7fa640 1 -- 172.21.15.156:0/291717484 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f4aac019880 con 0x7f4ac00b4d60 2024-03-25T18:17:41.960 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.959+0000 7f4ac4ec3640 10 monclient: get_auth_request con 0x7f4aa406e0f0 auth_method 0 2024-03-25T18:17:41.960 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.959+0000 7f4abd7fa640 10 client.4651.objecter ms_handle_connect 0x7f4aa406e0f0 2024-03-25T18:17:41.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.961+0000 7f4abd7fa640 1 -- 172.21.15.156:0/291717484 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f4aac053560 con 0x7f4ac00b4d60 2024-03-25T18:17:41.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.961+0000 7f4abd7fa640 10 client.4651.objecter ms_dispatch 0x7f4ac0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:41.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.961+0000 7f4abd7fa640 3 client.4651.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:41.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.961+0000 7f4abd7fa640 3 client.4651.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:41.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.961+0000 7f4abd7fa640 20 client.4651.objecter dump_active .. 0 homeless 2024-03-25T18:17:41.965 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.964+0000 7f4abd7fa640 1 -- 172.21.15.156:0/291717484 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4aac04eea0 con 0x7f4ac00b4d60 2024-03-25T18:17:41.965 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.964+0000 7f4abd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:41.965 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:41.964+0000 7f4abd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:42.219 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:42.218+0000 7f4ac7127640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:42.219 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:42.218+0000 7f4ac7127640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:42.219 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:42.218+0000 7f4ac7127640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:17:42.219 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:42.218+0000 7f4ac7127640 1 -- 172.21.15.156:0/291717484 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ac0139360 con 0x7f4ac00b4d60 2024-03-25T18:17:42.265 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:42.264+0000 7f4abd7fa640 1 -- 172.21.15.156:0/291717484 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f4ac0139360 con 0x7f4ac00b4d60 2024-03-25T18:17:42.265 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:42.264+0000 7f4abd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:42.266 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:42.264+0000 7f4abd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:42.266 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:17:42.266 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:42.268 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:42.267+0000 7f4ac7127640 10 librados: watch_flush enter 2024-03-25T18:17:42.268 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:42.267+0000 7f4ac7127640 10 librados: watch_flush exit 2024-03-25T18:17:42.268 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:42.267+0000 7f4ac7127640 20 client.4651.objecter shutdown clearing up homeless session... 2024-03-25T18:17:42.269 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:42.267+0000 7f4ac7127640 10 client.4651.objecter successfully canceled tick 2024-03-25T18:17:42.269 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:42.267+0000 7f4ac7127640 1 -- 172.21.15.156:0/291717484 >> v1:172.21.15.194:6800/2946427713 conn(0x7f4aa406e0f0 legacy=0x7f4aa40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:42.269 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:42.267+0000 7f4ac7127640 10 monclient: shutdown 2024-03-25T18:17:42.269 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:42.267+0000 7f4ac7127640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:42.269 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:42.267+0000 7f4ac7127640 1 -- 172.21.15.156:0/291717484 >> v1:172.21.15.194:6789/0 conn(0x7f4ac00b4d60 legacy=0x7f4ac01dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:42.269 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:42.267+0000 7f4ac7127640 1 -- 172.21.15.156:0/291717484 shutdown_connections 2024-03-25T18:17:42.269 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:42.267+0000 7f4ac7127640 1 -- 172.21.15.156:0/291717484 >> 172.21.15.156:0/291717484 conn(0x7f4ac00617a0 msgr2=0x7f4ac00b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:42.269 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:42.267+0000 7f4ac7127640 1 -- 172.21.15.156:0/291717484 shutdown_connections 2024-03-25T18:17:42.269 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:42.267+0000 7f4ac7127640 1 -- 172.21.15.156:0/291717484 wait complete. 2024-03-25T18:17:42.269 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:42.267+0000 7f4ac7127640 1 librados: shutdown 2024-03-25T18:17:42.278 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:49.085 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:49.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.198+0000 7f38fd8ba640 10 monclient: get_monmap_and_config 2024-03-25T18:17:49.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.198+0000 7f38fd8ba640 10 monclient: build_initial_monmap 2024-03-25T18:17:49.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.198+0000 7f38fd8ba640 10 monclient: monmap: 2024-03-25T18:17:49.200 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:17:49.200 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:49.200 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:17:49.199782+0000 2024-03-25T18:17:49.200 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:17:49.199782+0000 2024-03-25T18:17:49.200 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:49.200 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:49.200 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:17:49.201 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:17:49.201 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:17:49.201 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:49.202 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.199+0000 7f38fd8ba640 1 Processor -- start 2024-03-25T18:17:49.202 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.199+0000 7f38fd8ba640 1 -- start start 2024-03-25T18:17:49.202 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.199+0000 7f38fd8ba640 10 monclient: init 2024-03-25T18:17:49.202 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.200+0000 7f38fd8ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:49.202 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.200+0000 7f38fd8ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:49.202 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.200+0000 7f38fd8ba640 10 monclient(hunting): picked mon.noname-a con 0x7f38f80a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:17:49.202 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.200+0000 7f38fd8ba640 10 monclient(hunting): picked mon.noname-b con 0x7f38f813f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:17:49.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.200+0000 7f38fd8ba640 10 monclient(hunting): picked mon.noname-c con 0x7f38f81430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:17:49.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.200+0000 7f38fd8ba640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f8142520 con 0x7f38f80a98f0 2024-03-25T18:17:49.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.200+0000 7f38fd8ba640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f80b28a0 con 0x7f38f81430c0 2024-03-25T18:17:49.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.200+0000 7f38fd8ba640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f80b2a30 con 0x7f38f813f710 2024-03-25T18:17:49.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.200+0000 7f38fd8ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:49.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.200+0000 7f38fd8ba640 10 monclient(hunting): authenticate will time out at 7550.511527s 2024-03-25T18:17:49.221 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.220+0000 7f38f67fc640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f38f813f710 0x7f38f81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:51740/0 (socket says 172.21.15.156:51740) 2024-03-25T18:17:49.221 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.220+0000 7f38f67fc640 1 -- 172.21.15.156:0/3293343390 learned_addr learned my addr 172.21.15.156:0/3293343390 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:17:49.234 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.233+0000 7f38f5ffb640 1 -- 172.21.15.156:0/3293343390 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 316527633 0 0) 0x7f38f80b2a30 con 0x7f38f813f710 2024-03-25T18:17:49.235 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.233+0000 7f38f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:49.235 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.233+0000 7f38f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:49.235 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.233+0000 7f38f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:49.235 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.233+0000 7f38f5ffb640 10 monclient(hunting): my global_id is 4658 2024-03-25T18:17:49.235 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.233+0000 7f38f5ffb640 1 -- 172.21.15.156:0/3293343390 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38e0003520 con 0x7f38f813f710 2024-03-25T18:17:49.243 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.242+0000 7f38f5ffb640 1 -- 172.21.15.156:0/3293343390 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3271192595 0 0) 0x7f38f80b28a0 con 0x7f38f81430c0 2024-03-25T18:17:49.244 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.243+0000 7f38f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:49.244 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.243+0000 7f38f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:49.244 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.243+0000 7f38f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:49.244 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.243+0000 7f38f5ffb640 10 monclient(hunting): my global_id is 4654 2024-03-25T18:17:49.244 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.243+0000 7f38f5ffb640 1 -- 172.21.15.156:0/3293343390 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38f80b2a30 con 0x7f38f81430c0 2024-03-25T18:17:49.248 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.247+0000 7f38f5ffb640 1 -- 172.21.15.156:0/3293343390 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 36476450 0 0) 0x7f38e0003520 con 0x7f38f813f710 2024-03-25T18:17:49.248 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.247+0000 7f38f5ffb640 1 -- 172.21.15.156:0/3293343390 >> v1:172.21.15.194:6789/0 conn(0x7f38f81430c0 legacy=0x7f38f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:49.248 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.247+0000 7f38f5ffb640 1 -- 172.21.15.156:0/3293343390 >> v1:172.21.15.156:6789/0 conn(0x7f38f80a98f0 legacy=0x7f38f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:49.248 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.247+0000 7f38f5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:17:49.248 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.247+0000 7f38f5ffb640 1 monclient: found mon.noname-b 2024-03-25T18:17:49.248 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.247+0000 7f38f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:49.248 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.247+0000 7f38f5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.156:6790/0 2024-03-25T18:17:49.248 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.247+0000 7f38f5ffb640 1 -- 172.21.15.156:0/3293343390 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f80b44c0 con 0x7f38f813f710 2024-03-25T18:17:49.248 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.247+0000 7f38f5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:17:49.249 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.247+0000 7f38f5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:17:49.249 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.247+0000 7f38f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:49.249 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.247+0000 7f38f5ffb640 1 -- 172.21.15.156:0/3293343390 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f38ec001690 con 0x7f38f813f710 2024-03-25T18:17:49.249 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.247+0000 7f38f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:49.249 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.247+0000 7f38f5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:49.249 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.247+0000 7f38f5ffb640 10 monclient: dump: 2024-03-25T18:17:49.249 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:49.249 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:49.249 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:49.249 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:49.249 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:49.249 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:49.249 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:49.249 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:49.249 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:49.249 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:49.250 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.247+0000 7f38fd8ba640 5 monclient: authenticate success, global_id 4658 2024-03-25T18:17:49.250 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.247+0000 7f38fd8ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:49.261 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.260+0000 7f38f5ffb640 1 -- 172.21.15.156:0/3293343390 <== mon.2 v1:172.21.15.156:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38ec0017f0 con 0x7f38f813f710 2024-03-25T18:17:49.261 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.260+0000 7f38f5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:49.261 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.260+0000 7f38f5ffb640 1 -- 172.21.15.156:0/3293343390 <== mon.2 v1:172.21.15.156:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f38ec001c00 con 0x7f38f813f710 2024-03-25T18:17:49.261 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.260+0000 7f38f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:49.261 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.260+0000 7f38f5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:49.262 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.260+0000 7f38f5ffb640 10 monclient: dump: 2024-03-25T18:17:49.262 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:49.262 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:49.262 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:49.262 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:49.262 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:49.262 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:49.262 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:49.262 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:49.262 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:49.262 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:49.262 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.260+0000 7f38fd8ba640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:49.262 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.260+0000 7f38fd8ba640 4 set_mon_vals no callback set 2024-03-25T18:17:49.264 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.261+0000 7f38fd8ba640 10 monclient: shutdown 2024-03-25T18:17:49.264 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.261+0000 7f38fd8ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:49.264 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.261+0000 7f38fd8ba640 1 -- 172.21.15.156:0/3293343390 >> v1:172.21.15.156:6790/0 conn(0x7f38f813f710 legacy=0x7f38f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:49.264 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.261+0000 7f38fd8ba640 1 -- 172.21.15.156:0/3293343390 shutdown_connections 2024-03-25T18:17:49.264 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.261+0000 7f38fd8ba640 1 -- 172.21.15.156:0/3293343390 >> 172.21.15.156:0/3293343390 conn(0x7f38f80b7ea0 msgr2=0x7f38f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:49.264 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.261+0000 7f38fd8ba640 1 -- 172.21.15.156:0/3293343390 shutdown_connections 2024-03-25T18:17:49.264 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.261+0000 7f38fd8ba640 1 -- 172.21.15.156:0/3293343390 wait complete. 2024-03-25T18:17:49.264 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.262+0000 7f38fd8ba640 10 monclient: build_initial_monmap 2024-03-25T18:17:49.264 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.262+0000 7f38fd8ba640 10 monclient: monmap: 2024-03-25T18:17:49.265 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:17:49.265 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:49.265 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:17:49.265 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:17:49.265 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:49.265 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:17:49.265 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:17:49.265 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:17:49.265 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:17:49.265 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:49.265 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.262+0000 7f38fd8ba640 1 librados: starting msgr at 2024-03-25T18:17:49.265 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.262+0000 7f38fd8ba640 1 librados: starting objecter 2024-03-25T18:17:49.265 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.263+0000 7f38fd8ba640 1 Processor -- start 2024-03-25T18:17:49.265 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.263+0000 7f38fd8ba640 1 -- start start 2024-03-25T18:17:49.266 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.263+0000 7f38fd8ba640 1 librados: setting wanted keys 2024-03-25T18:17:49.266 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.263+0000 7f38fd8ba640 1 librados: calling monclient init 2024-03-25T18:17:49.266 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.263+0000 7f38fd8ba640 10 monclient: init 2024-03-25T18:17:49.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.264+0000 7f38fd8ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:49.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.264+0000 7f38fd8ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:49.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.264+0000 7f38fd8ba640 10 monclient(hunting): picked mon.noname-a con 0x7f38f80a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:17:49.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.264+0000 7f38fd8ba640 10 monclient(hunting): picked mon.noname-c con 0x7f38f813f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:17:49.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.264+0000 7f38fd8ba640 10 monclient(hunting): picked mon.noname-b con 0x7f38f81430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:17:49.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.264+0000 7f38fd8ba640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f80abdc0 con 0x7f38f80a98f0 2024-03-25T18:17:49.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.264+0000 7f38fd8ba640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f81d7730 con 0x7f38f81430c0 2024-03-25T18:17:49.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.264+0000 7f38fd8ba640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f8138f10 con 0x7f38f813f710 2024-03-25T18:17:49.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.264+0000 7f38fd8ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:49.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.264+0000 7f38fd8ba640 10 monclient(hunting): authenticate will time out at 7550.574933s 2024-03-25T18:17:49.277 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.276+0000 7f38f77fe640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f38f81430c0 0x7f38f81d4860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:42590/0 (socket says 172.21.15.156:42590) 2024-03-25T18:17:49.277 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.276+0000 7f38f77fe640 1 -- 172.21.15.156:0/2646921996 learned_addr learned my addr 172.21.15.156:0/2646921996 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:17:49.277 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.276+0000 7f38dffff640 10 client.?.objecter ms_handle_connect 0x7f38f81430c0 2024-03-25T18:17:49.277 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.277+0000 7f38dffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:49.278 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.277+0000 7f38dffff640 10 client.?.objecter ms_handle_connect 0x7f38f80a98f0 2024-03-25T18:17:49.278 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.277+0000 7f38dffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:49.284 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.283+0000 7f38dffff640 1 -- 172.21.15.156:0/2646921996 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2634365730 0 0) 0x7f38f80abdc0 con 0x7f38f80a98f0 2024-03-25T18:17:49.284 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.283+0000 7f38dffff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:49.285 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.283+0000 7f38dffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:49.285 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.283+0000 7f38dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:49.285 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.283+0000 7f38dffff640 10 monclient(hunting): my global_id is 4653 2024-03-25T18:17:49.285 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.283+0000 7f38dffff640 1 -- 172.21.15.156:0/2646921996 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38e4004eb0 con 0x7f38f80a98f0 2024-03-25T18:17:49.285 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.284+0000 7f38dffff640 1 -- 172.21.15.156:0/2646921996 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 150280337 0 0) 0x7f38f81d7730 con 0x7f38f81430c0 2024-03-25T18:17:49.286 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.284+0000 7f38dffff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:49.286 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.284+0000 7f38dffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:49.286 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.284+0000 7f38dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:49.286 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.284+0000 7f38dffff640 10 monclient(hunting): my global_id is 4657 2024-03-25T18:17:49.286 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.284+0000 7f38dffff640 1 -- 172.21.15.156:0/2646921996 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38f80abdc0 con 0x7f38f81430c0 2024-03-25T18:17:49.289 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.288+0000 7f38dffff640 10 client.?.objecter ms_handle_connect 0x7f38f813f710 2024-03-25T18:17:49.289 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.288+0000 7f38dffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:49.293 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.292+0000 7f38dffff640 1 -- 172.21.15.156:0/2646921996 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3614191815 0 0) 0x7f38e4004eb0 con 0x7f38f80a98f0 2024-03-25T18:17:49.293 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.292+0000 7f38dffff640 1 -- 172.21.15.156:0/2646921996 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38f81d7730 con 0x7f38f80a98f0 2024-03-25T18:17:49.293 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.292+0000 7f38dffff640 1 -- 172.21.15.156:0/2646921996 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f38e80018c0 con 0x7f38f80a98f0 2024-03-25T18:17:49.293 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.292+0000 7f38dffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:49.293 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.292+0000 7f38dffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:49.293 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.292+0000 7f38dffff640 10 monclient(hunting): dump: 2024-03-25T18:17:49.293 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:49.293 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:49.294 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:49.294 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:49.294 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:49.294 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:49.294 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:49.294 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:49.294 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:49.294 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:49.294 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.292+0000 7f38dffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:49.301 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.299+0000 7f38dffff640 1 -- 172.21.15.156:0/2646921996 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1396669027 0 0) 0x7f38f80abdc0 con 0x7f38f81430c0 2024-03-25T18:17:49.301 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.300+0000 7f38dffff640 1 -- 172.21.15.156:0/2646921996 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38e4004eb0 con 0x7f38f81430c0 2024-03-25T18:17:49.301 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.300+0000 7f38dffff640 1 -- 172.21.15.156:0/2646921996 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f38f0001c50 con 0x7f38f81430c0 2024-03-25T18:17:49.301 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.300+0000 7f38dffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:49.301 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.300+0000 7f38dffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:49.301 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.300+0000 7f38dffff640 10 monclient(hunting): dump: 2024-03-25T18:17:49.301 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:49.301 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:49.301 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:49.301 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:49.301 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:49.301 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:49.301 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:49.302 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:49.302 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:49.302 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:49.304 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.302+0000 7f38dffff640 1 -- 172.21.15.156:0/2646921996 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3923146493 0 0) 0x7f38f81d7730 con 0x7f38f80a98f0 2024-03-25T18:17:49.304 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.302+0000 7f38dffff640 1 -- 172.21.15.156:0/2646921996 >> v1:172.21.15.156:6790/0 conn(0x7f38f813f710 legacy=0x7f38f81d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:49.304 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.302+0000 7f38dffff640 1 -- 172.21.15.156:0/2646921996 >> v1:172.21.15.194:6789/0 conn(0x7f38f81430c0 legacy=0x7f38f81d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:49.304 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.302+0000 7f38dffff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:49.304 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.302+0000 7f38dffff640 1 monclient: found mon.a 2024-03-25T18:17:49.304 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.302+0000 7f38dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:49.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.302+0000 7f38dffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:17:49.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.302+0000 7f38dffff640 1 -- 172.21.15.156:0/2646921996 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f81390a0 con 0x7f38f80a98f0 2024-03-25T18:17:49.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.303+0000 7f38dffff640 10 monclient: _finish_auth 0 2024-03-25T18:17:49.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.303+0000 7f38dffff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:49.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.303+0000 7f38dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:49.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.303+0000 7f38fd8ba640 5 monclient: authenticate success, global_id 4653 2024-03-25T18:17:49.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.303+0000 7f38fd8ba640 10 monclient: _renew_subs 2024-03-25T18:17:49.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.303+0000 7f38fd8ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:17:49.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.303+0000 7f38fd8ba640 1 -- 172.21.15.156:0/2646921996 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38f81d8320 con 0x7f38f80a98f0 2024-03-25T18:17:49.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.303+0000 7f38fd8ba640 10 client.4653.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:49.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.303+0000 7f38fd8ba640 10 monclient: _renew_subs 2024-03-25T18:17:49.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.303+0000 7f38fd8ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:17:49.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.303+0000 7f38fd8ba640 1 -- 172.21.15.156:0/2646921996 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38f81d43c0 con 0x7f38f80a98f0 2024-03-25T18:17:49.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.303+0000 7f38fd8ba640 1 librados: init done 2024-03-25T18:17:49.306 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.304+0000 7f38fd8ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:49.306 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.304+0000 7f38fd8ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:49.306 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.304+0000 7f38fd8ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:17:49.306 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.304+0000 7f38fd8ba640 1 -- 172.21.15.156:0/2646921996 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38f80627b0 con 0x7f38f80a98f0 2024-03-25T18:17:49.606 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.605+0000 7f38dffff640 1 -- 172.21.15.156:0/2646921996 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38e80035f0 con 0x7f38f80a98f0 2024-03-25T18:17:49.606 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.605+0000 7f38dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:49.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.605+0000 7f38dffff640 1 -- 172.21.15.156:0/2646921996 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f38e8003900 con 0x7f38f80a98f0 2024-03-25T18:17:49.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.605+0000 7f38dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:49.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.605+0000 7f38dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:49.607 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.605+0000 7f38dffff640 10 monclient: dump: 2024-03-25T18:17:49.607 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:49.607 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:49.607 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:49.607 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:49.607 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:49.607 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:49.607 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:49.607 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:49.607 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:49.608 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:49.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.605+0000 7f38f57fa640 4 set_mon_vals no callback set 2024-03-25T18:17:49.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.606+0000 7f38dffff640 1 -- 172.21.15.156:0/2646921996 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f38e80198d0 con 0x7f38f80a98f0 2024-03-25T18:17:49.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.606+0000 7f38f67fc640 10 monclient: get_auth_request con 0x7f38e406de40 auth_method 0 2024-03-25T18:17:49.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.607+0000 7f38dffff640 10 client.4653.objecter ms_handle_connect 0x7f38e406de40 2024-03-25T18:17:49.614 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.612+0000 7f38dffff640 1 -- 172.21.15.156:0/2646921996 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f38e8053570 con 0x7f38f80a98f0 2024-03-25T18:17:49.614 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.612+0000 7f38dffff640 10 client.4653.objecter ms_dispatch 0x7f38f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:49.614 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.612+0000 7f38dffff640 3 client.4653.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:49.614 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.612+0000 7f38dffff640 3 client.4653.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:49.614 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.612+0000 7f38dffff640 20 client.4653.objecter dump_active .. 0 homeless 2024-03-25T18:17:49.614 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.612+0000 7f38dffff640 1 -- 172.21.15.156:0/2646921996 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f38e804f080 con 0x7f38f80a98f0 2024-03-25T18:17:49.614 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.612+0000 7f38dffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:49.614 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.612+0000 7f38dffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:49.870 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.869+0000 7f38fd8ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:49.870 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.869+0000 7f38fd8ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:49.871 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.869+0000 7f38fd8ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:17:49.871 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.869+0000 7f38fd8ba640 1 -- 172.21.15.156:0/2646921996 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38f8069f40 con 0x7f38f80a98f0 2024-03-25T18:17:49.902 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.901+0000 7f38dffff640 1 -- 172.21.15.156:0/2646921996 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f38e8050fb0 con 0x7f38f80a98f0 2024-03-25T18:17:49.903 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.901+0000 7f38dffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:49.903 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.901+0000 7f38dffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:49.903 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:17:49.903 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:49.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.904+0000 7f38fd8ba640 10 librados: watch_flush enter 2024-03-25T18:17:49.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.904+0000 7f38fd8ba640 10 librados: watch_flush exit 2024-03-25T18:17:49.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.904+0000 7f38fd8ba640 20 client.4653.objecter shutdown clearing up homeless session... 2024-03-25T18:17:49.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.904+0000 7f38fd8ba640 10 client.4653.objecter successfully canceled tick 2024-03-25T18:17:49.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.904+0000 7f38fd8ba640 1 -- 172.21.15.156:0/2646921996 >> v1:172.21.15.194:6800/2946427713 conn(0x7f38e406de40 legacy=0x7f38e4070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:49.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.904+0000 7f38fd8ba640 10 monclient: shutdown 2024-03-25T18:17:49.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.904+0000 7f38fd8ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:49.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.904+0000 7f38fd8ba640 1 -- 172.21.15.156:0/2646921996 >> v1:172.21.15.156:6789/0 conn(0x7f38f80a98f0 legacy=0x7f38f8138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:49.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.904+0000 7f38fd8ba640 1 -- 172.21.15.156:0/2646921996 shutdown_connections 2024-03-25T18:17:49.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.904+0000 7f38fd8ba640 1 -- 172.21.15.156:0/2646921996 >> 172.21.15.156:0/2646921996 conn(0x7f38f80b7ea0 msgr2=0x7f38f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:49.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.904+0000 7f38fd8ba640 1 -- 172.21.15.156:0/2646921996 shutdown_connections 2024-03-25T18:17:49.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.904+0000 7f38fd8ba640 1 -- 172.21.15.156:0/2646921996 wait complete. 2024-03-25T18:17:49.907 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:49.904+0000 7f38fd8ba640 1 librados: shutdown 2024-03-25T18:17:49.917 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:56.724 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:56.839 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.838+0000 7ffa36710640 10 monclient: get_monmap_and_config 2024-03-25T18:17:56.839 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.838+0000 7ffa36710640 10 monclient: build_initial_monmap 2024-03-25T18:17:56.839 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.838+0000 7ffa36710640 10 monclient: monmap: 2024-03-25T18:17:56.839 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:17:56.839 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:56.839 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:17:56.839175+0000 2024-03-25T18:17:56.840 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:17:56.839175+0000 2024-03-25T18:17:56.840 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:56.840 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:56.840 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:17:56.840 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:17:56.840 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:17:56.840 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:56.841 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.839+0000 7ffa36710640 1 Processor -- start 2024-03-25T18:17:56.842 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.839+0000 7ffa36710640 1 -- start start 2024-03-25T18:17:56.842 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.839+0000 7ffa36710640 10 monclient: init 2024-03-25T18:17:56.842 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.839+0000 7ffa36710640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:56.842 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.839+0000 7ffa36710640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:56.842 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.839+0000 7ffa36710640 10 monclient(hunting): picked mon.noname-b con 0x7ffa300b4d60 addr v1:172.21.15.156:6790/0 2024-03-25T18:17:56.842 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.839+0000 7ffa36710640 10 monclient(hunting): picked mon.noname-c con 0x7ffa300b1730 addr v1:172.21.15.194:6789/0 2024-03-25T18:17:56.842 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.839+0000 7ffa36710640 10 monclient(hunting): picked mon.noname-a con 0x7ffa3014be90 addr v1:172.21.15.156:6789/0 2024-03-25T18:17:56.842 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.839+0000 7ffa36710640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa300af940 con 0x7ffa3014be90 2024-03-25T18:17:56.842 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.839+0000 7ffa36710640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa300b0a70 con 0x7ffa300b1730 2024-03-25T18:17:56.842 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.839+0000 7ffa36710640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa300b0d50 con 0x7ffa300b4d60 2024-03-25T18:17:56.843 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.839+0000 7ffa36710640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:56.843 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.839+0000 7ffa36710640 10 monclient(hunting): authenticate will time out at 7558.150565s 2024-03-25T18:17:56.851 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.850+0000 7ffa2f7fe640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7ffa300b1730 0x7ffa300b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:46874/0 (socket says 172.21.15.156:46874) 2024-03-25T18:17:56.851 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.850+0000 7ffa2f7fe640 1 -- 172.21.15.156:0/2244412263 learned_addr learned my addr 172.21.15.156:0/2244412263 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:17:56.860 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.859+0000 7ffa2effd640 1 -- 172.21.15.156:0/2244412263 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1256774975 0 0) 0x7ffa300b0a70 con 0x7ffa300b1730 2024-03-25T18:17:56.861 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.859+0000 7ffa2effd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:56.861 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.859+0000 7ffa2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:56.861 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.859+0000 7ffa2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:56.861 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.859+0000 7ffa2effd640 10 monclient(hunting): my global_id is 4660 2024-03-25T18:17:56.861 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.859+0000 7ffa2effd640 1 -- 172.21.15.156:0/2244412263 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa18003520 con 0x7ffa300b1730 2024-03-25T18:17:56.872 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.871+0000 7ffa2effd640 1 -- 172.21.15.156:0/2244412263 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3913367293 0 0) 0x7ffa18003520 con 0x7ffa300b1730 2024-03-25T18:17:56.873 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.871+0000 7ffa2effd640 1 -- 172.21.15.156:0/2244412263 >> v1:172.21.15.156:6790/0 conn(0x7ffa300b4d60 legacy=0x7ffa3013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:56.873 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.871+0000 7ffa2effd640 1 -- 172.21.15.156:0/2244412263 >> v1:172.21.15.156:6789/0 conn(0x7ffa3014be90 legacy=0x7ffa3014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:56.873 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.871+0000 7ffa2effd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:56.873 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.871+0000 7ffa2effd640 1 monclient: found mon.noname-c 2024-03-25T18:17:56.873 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.871+0000 7ffa2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:56.873 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.871+0000 7ffa2effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:17:56.873 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.871+0000 7ffa2effd640 1 -- 172.21.15.156:0/2244412263 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa300b1030 con 0x7ffa300b1730 2024-03-25T18:17:56.873 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.871+0000 7ffa2effd640 10 monclient: _finish_auth 0 2024-03-25T18:17:56.873 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.871+0000 7ffa2effd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:56.873 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.871+0000 7ffa2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:56.873 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.871+0000 7ffa2effd640 1 -- 172.21.15.156:0/2244412263 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7ffa24001690 con 0x7ffa300b1730 2024-03-25T18:17:56.874 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.871+0000 7ffa2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:56.874 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.871+0000 7ffa2effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:56.874 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.871+0000 7ffa2effd640 10 monclient: dump: 2024-03-25T18:17:56.874 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:56.874 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:56.874 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:56.874 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:56.874 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:56.874 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:56.874 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:56.874 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:56.875 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:56.875 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:56.875 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.871+0000 7ffa36710640 5 monclient: authenticate success, global_id 4660 2024-03-25T18:17:56.875 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.871+0000 7ffa36710640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:56.878 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.876+0000 7ffa2effd640 1 -- 172.21.15.156:0/2244412263 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa240017f0 con 0x7ffa300b1730 2024-03-25T18:17:56.878 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.876+0000 7ffa2effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:56.878 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.876+0000 7ffa2effd640 1 -- 172.21.15.156:0/2244412263 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7ffa24001c00 con 0x7ffa300b1730 2024-03-25T18:17:56.878 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.876+0000 7ffa2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:56.878 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.876+0000 7ffa2effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:56.878 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.876+0000 7ffa2effd640 10 monclient: dump: 2024-03-25T18:17:56.878 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:56.878 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:56.878 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:56.879 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:56.879 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:56.879 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:56.879 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:56.879 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:56.879 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:56.879 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:56.879 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.876+0000 7ffa36710640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:56.879 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.876+0000 7ffa36710640 4 set_mon_vals no callback set 2024-03-25T18:17:56.882 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.877+0000 7ffa36710640 10 monclient: shutdown 2024-03-25T18:17:56.882 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.877+0000 7ffa36710640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:56.882 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.877+0000 7ffa36710640 1 -- 172.21.15.156:0/2244412263 >> v1:172.21.15.194:6789/0 conn(0x7ffa300b1730 legacy=0x7ffa300b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:56.882 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.877+0000 7ffa36710640 1 -- 172.21.15.156:0/2244412263 shutdown_connections 2024-03-25T18:17:56.882 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.877+0000 7ffa36710640 1 -- 172.21.15.156:0/2244412263 >> 172.21.15.156:0/2244412263 conn(0x7ffa300617a0 msgr2=0x7ffa30061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:56.882 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.877+0000 7ffa36710640 1 -- 172.21.15.156:0/2244412263 shutdown_connections 2024-03-25T18:17:56.883 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.877+0000 7ffa36710640 1 -- 172.21.15.156:0/2244412263 wait complete. 2024-03-25T18:17:56.883 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.878+0000 7ffa36710640 10 monclient: build_initial_monmap 2024-03-25T18:17:56.883 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.878+0000 7ffa36710640 10 monclient: monmap: 2024-03-25T18:17:56.883 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:17:56.883 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:56.883 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:17:56.883 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:17:56.883 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:56.883 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:17:56.883 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:17:56.883 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:17:56.884 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:17:56.884 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:56.884 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.879+0000 7ffa36710640 1 librados: starting msgr at 2024-03-25T18:17:56.884 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.879+0000 7ffa36710640 1 librados: starting objecter 2024-03-25T18:17:56.884 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.879+0000 7ffa36710640 1 Processor -- start 2024-03-25T18:17:56.884 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.879+0000 7ffa36710640 1 -- start start 2024-03-25T18:17:56.884 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.879+0000 7ffa36710640 1 librados: setting wanted keys 2024-03-25T18:17:56.884 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.879+0000 7ffa36710640 1 librados: calling monclient init 2024-03-25T18:17:56.884 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.879+0000 7ffa36710640 10 monclient: init 2024-03-25T18:17:56.884 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.879+0000 7ffa36710640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:56.885 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.879+0000 7ffa36710640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:17:56.885 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.879+0000 7ffa36710640 10 monclient(hunting): picked mon.noname-b con 0x7ffa300b1730 addr v1:172.21.15.194:6789/0 2024-03-25T18:17:56.885 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.879+0000 7ffa36710640 10 monclient(hunting): picked mon.noname-c con 0x7ffa300b4d60 addr v1:172.21.15.156:6790/0 2024-03-25T18:17:56.885 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.879+0000 7ffa36710640 10 monclient(hunting): picked mon.noname-a con 0x7ffa3014be90 addr v1:172.21.15.156:6789/0 2024-03-25T18:17:56.885 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.879+0000 7ffa36710640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa301d83d0 con 0x7ffa3014be90 2024-03-25T18:17:56.885 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.880+0000 7ffa36710640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa301d77a0 con 0x7ffa300b1730 2024-03-25T18:17:56.885 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.880+0000 7ffa36710640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa30138270 con 0x7ffa300b4d60 2024-03-25T18:17:56.885 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.880+0000 7ffa36710640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:56.885 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.880+0000 7ffa36710640 10 monclient(hunting): authenticate will time out at 7558.190834s 2024-03-25T18:17:56.886 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.885+0000 7ffa34cad640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7ffa3014be90 0x7ffa30142d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:34916/0 (socket says 172.21.15.156:34916) 2024-03-25T18:17:56.886 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.885+0000 7ffa34cad640 1 -- 172.21.15.156:0/1216647053 learned_addr learned my addr 172.21.15.156:0/1216647053 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:17:56.887 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.886+0000 7ffa2cff9640 10 client.?.objecter ms_handle_connect 0x7ffa300b4d60 2024-03-25T18:17:56.887 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.886+0000 7ffa2cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:56.888 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.887+0000 7ffa2cff9640 10 client.?.objecter ms_handle_connect 0x7ffa3014be90 2024-03-25T18:17:56.888 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.887+0000 7ffa2cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:56.900 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.898+0000 7ffa2cff9640 1 -- 172.21.15.156:0/1216647053 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3517384928 0 0) 0x7ffa30138270 con 0x7ffa300b4d60 2024-03-25T18:17:56.900 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.898+0000 7ffa2cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:56.900 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.898+0000 7ffa2cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:56.900 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.898+0000 7ffa2cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:56.900 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.898+0000 7ffa2cff9640 10 monclient(hunting): my global_id is 4667 2024-03-25T18:17:56.900 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.898+0000 7ffa2cff9640 1 -- 172.21.15.156:0/1216647053 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa1c004eb0 con 0x7ffa300b4d60 2024-03-25T18:17:56.901 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.900+0000 7ffa2cff9640 10 client.?.objecter ms_handle_connect 0x7ffa300b1730 2024-03-25T18:17:56.901 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.900+0000 7ffa2cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:56.903 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.902+0000 7ffa2cff9640 1 -- 172.21.15.156:0/1216647053 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700060681 0 0) 0x7ffa301d83d0 con 0x7ffa3014be90 2024-03-25T18:17:56.903 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.902+0000 7ffa2cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:56.903 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.902+0000 7ffa2cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:56.903 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.902+0000 7ffa2cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:56.903 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.902+0000 7ffa2cff9640 10 monclient(hunting): my global_id is 4659 2024-03-25T18:17:56.903 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.902+0000 7ffa2cff9640 1 -- 172.21.15.156:0/1216647053 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa30138270 con 0x7ffa3014be90 2024-03-25T18:17:56.913 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.912+0000 7ffa2cff9640 1 -- 172.21.15.156:0/1216647053 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1234800331 0 0) 0x7ffa30138270 con 0x7ffa3014be90 2024-03-25T18:17:56.913 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.912+0000 7ffa2cff9640 1 -- 172.21.15.156:0/1216647053 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa301d83d0 con 0x7ffa3014be90 2024-03-25T18:17:56.914 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.912+0000 7ffa2cff9640 1 -- 172.21.15.156:0/1216647053 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7ffa28001c10 con 0x7ffa3014be90 2024-03-25T18:17:56.914 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.912+0000 7ffa2cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:56.914 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.912+0000 7ffa2cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:56.914 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.912+0000 7ffa2cff9640 10 monclient(hunting): dump: 2024-03-25T18:17:56.914 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:56.914 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:56.914 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:56.914 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:56.914 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:56.914 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:56.914 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:56.915 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:56.915 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:56.915 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:56.915 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.912+0000 7ffa2cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:56.926 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.924+0000 7ffa2cff9640 1 -- 172.21.15.156:0/1216647053 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3339729543 0 0) 0x7ffa301d83d0 con 0x7ffa3014be90 2024-03-25T18:17:56.926 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.924+0000 7ffa2cff9640 1 -- 172.21.15.156:0/1216647053 >> v1:172.21.15.156:6790/0 conn(0x7ffa300b4d60 legacy=0x7ffa301405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:56.927 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.924+0000 7ffa2cff9640 1 -- 172.21.15.156:0/1216647053 >> v1:172.21.15.194:6789/0 conn(0x7ffa300b1730 legacy=0x7ffa301d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:56.927 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.924+0000 7ffa2cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:17:56.927 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.924+0000 7ffa2cff9640 1 monclient: found mon.a 2024-03-25T18:17:56.927 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.924+0000 7ffa2cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:56.927 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.924+0000 7ffa2cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:17:56.927 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.924+0000 7ffa2cff9640 1 -- 172.21.15.156:0/1216647053 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa30140d00 con 0x7ffa3014be90 2024-03-25T18:17:56.927 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.924+0000 7ffa2cff9640 10 monclient: _finish_auth 0 2024-03-25T18:17:56.927 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.924+0000 7ffa2cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:17:56.927 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.924+0000 7ffa2cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:56.927 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.924+0000 7ffa36710640 5 monclient: authenticate success, global_id 4659 2024-03-25T18:17:56.927 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.924+0000 7ffa36710640 10 monclient: _renew_subs 2024-03-25T18:17:56.928 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.924+0000 7ffa36710640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:17:56.928 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.924+0000 7ffa36710640 1 -- 172.21.15.156:0/1216647053 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa30138a80 con 0x7ffa3014be90 2024-03-25T18:17:56.928 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.924+0000 7ffa36710640 10 client.4659.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:56.928 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.924+0000 7ffa36710640 10 monclient: _renew_subs 2024-03-25T18:17:56.928 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.924+0000 7ffa36710640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:17:56.928 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.924+0000 7ffa36710640 1 -- 172.21.15.156:0/1216647053 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa30143640 con 0x7ffa3014be90 2024-03-25T18:17:56.928 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.924+0000 7ffa36710640 1 librados: init done 2024-03-25T18:17:56.928 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.925+0000 7ffa36710640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:56.928 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.925+0000 7ffa36710640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:56.928 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.925+0000 7ffa36710640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:17:56.928 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:56.925+0000 7ffa36710640 1 -- 172.21.15.156:0/1216647053 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa30140ef0 con 0x7ffa3014be90 2024-03-25T18:17:57.224 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.223+0000 7ffa2cff9640 1 -- 172.21.15.156:0/1216647053 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa28003990 con 0x7ffa3014be90 2024-03-25T18:17:57.225 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.223+0000 7ffa2cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:57.225 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.223+0000 7ffa2cff9640 1 -- 172.21.15.156:0/1216647053 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7ffa28002100 con 0x7ffa3014be90 2024-03-25T18:17:57.225 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.223+0000 7ffa2cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:57.225 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.223+0000 7ffa2cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:57.225 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.223+0000 7ffa2cff9640 10 monclient: dump: 2024-03-25T18:17:57.225 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:17:57.225 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:17:57.225 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:57.226 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:17:57.226 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:57.226 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:17:57.226 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:17:57.226 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:17:57.226 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:17:57.226 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:17:57.226 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.223+0000 7ffa2e7fc640 4 set_mon_vals no callback set 2024-03-25T18:17:57.227 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.224+0000 7ffa2cff9640 1 -- 172.21.15.156:0/1216647053 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7ffa28003c80 con 0x7ffa3014be90 2024-03-25T18:17:57.227 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.224+0000 7ffa2ffff640 10 monclient: get_auth_request con 0x7ffa1c06dd60 auth_method 0 2024-03-25T18:17:57.227 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.225+0000 7ffa2cff9640 10 client.4659.objecter ms_handle_connect 0x7ffa1c06dd60 2024-03-25T18:17:57.231 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.229+0000 7ffa2cff9640 1 -- 172.21.15.156:0/1216647053 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7ffa28054870 con 0x7ffa3014be90 2024-03-25T18:17:57.231 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.229+0000 7ffa2cff9640 10 client.4659.objecter ms_dispatch 0x7ffa30000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:57.231 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.229+0000 7ffa2cff9640 3 client.4659.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:57.231 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.229+0000 7ffa2cff9640 3 client.4659.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:57.231 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.229+0000 7ffa2cff9640 20 client.4659.objecter dump_active .. 0 homeless 2024-03-25T18:17:57.231 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.229+0000 7ffa2cff9640 1 -- 172.21.15.156:0/1216647053 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffa280501b0 con 0x7ffa3014be90 2024-03-25T18:17:57.231 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.229+0000 7ffa2cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:57.231 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.229+0000 7ffa2cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:57.479 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.477+0000 7ffa36710640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:57.479 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.477+0000 7ffa36710640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:57.479 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.477+0000 7ffa36710640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:17:57.479 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.477+0000 7ffa36710640 1 -- 172.21.15.156:0/1216647053 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa300afbd0 con 0x7ffa3014be90 2024-03-25T18:17:57.510 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.509+0000 7ffa2cff9640 1 -- 172.21.15.156:0/1216647053 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7ffa280522b0 con 0x7ffa3014be90 2024-03-25T18:17:57.510 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.509+0000 7ffa2cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:57.510 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.509+0000 7ffa2cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:57.511 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:17:57.511 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:57.514 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.511+0000 7ffa36710640 10 librados: watch_flush enter 2024-03-25T18:17:57.514 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.512+0000 7ffa36710640 10 librados: watch_flush exit 2024-03-25T18:17:57.514 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.512+0000 7ffa36710640 20 client.4659.objecter shutdown clearing up homeless session... 2024-03-25T18:17:57.514 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.512+0000 7ffa36710640 10 client.4659.objecter successfully canceled tick 2024-03-25T18:17:57.514 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.512+0000 7ffa36710640 1 -- 172.21.15.156:0/1216647053 >> v1:172.21.15.194:6800/2946427713 conn(0x7ffa1c06dd60 legacy=0x7ffa1c070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:57.514 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.512+0000 7ffa36710640 10 monclient: shutdown 2024-03-25T18:17:57.514 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.512+0000 7ffa36710640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:57.514 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.512+0000 7ffa36710640 1 -- 172.21.15.156:0/1216647053 >> v1:172.21.15.156:6789/0 conn(0x7ffa3014be90 legacy=0x7ffa30142d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:57.514 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.512+0000 7ffa36710640 1 -- 172.21.15.156:0/1216647053 shutdown_connections 2024-03-25T18:17:57.514 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.512+0000 7ffa36710640 1 -- 172.21.15.156:0/1216647053 >> 172.21.15.156:0/1216647053 conn(0x7ffa300617a0 msgr2=0x7ffa30061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:57.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.512+0000 7ffa36710640 1 -- 172.21.15.156:0/1216647053 shutdown_connections 2024-03-25T18:17:57.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.512+0000 7ffa36710640 1 -- 172.21.15.156:0/1216647053 wait complete. 2024-03-25T18:17:57.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:17:57.512+0000 7ffa36710640 1 librados: shutdown 2024-03-25T18:17:57.526 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:59.994 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:10:05.984 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:00.038 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:10:06.005 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:00.040 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:10:06.006 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:00.042 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:10:06.032 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:00.078 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:10:06.068 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:00.091 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:10:06.058 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:00.102 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:10:06.093 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:00.110 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:10:06.100 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:00.138 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:10:06.128 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:04.332 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:04.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.433+0000 7fa081405640 10 monclient: get_monmap_and_config 2024-03-25T18:18:04.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.433+0000 7fa081405640 10 monclient: build_initial_monmap 2024-03-25T18:18:04.435 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.434+0000 7fa081405640 10 monclient: monmap: 2024-03-25T18:18:04.435 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:18:04.435 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:04.435 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:18:04.434986+0000 2024-03-25T18:18:04.435 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:18:04.434986+0000 2024-03-25T18:18:04.435 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:04.436 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:04.436 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:18:04.436 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:18:04.436 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:18:04.436 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:04.437 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.434+0000 7fa081405640 1 Processor -- start 2024-03-25T18:18:04.437 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.434+0000 7fa081405640 1 -- start start 2024-03-25T18:18:04.437 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.434+0000 7fa081405640 10 monclient: init 2024-03-25T18:18:04.437 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.435+0000 7fa081405640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:04.437 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.435+0000 7fa081405640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:04.437 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.435+0000 7fa081405640 10 monclient(hunting): picked mon.noname-b con 0x7fa07c0b10d0 addr v1:172.21.15.156:6790/0 2024-03-25T18:18:04.437 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.435+0000 7fa081405640 10 monclient(hunting): picked mon.noname-c con 0x7fa07c0b4360 addr v1:172.21.15.194:6789/0 2024-03-25T18:18:04.438 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.435+0000 7fa081405640 10 monclient(hunting): picked mon.noname-a con 0x7fa07c0a9470 addr v1:172.21.15.156:6789/0 2024-03-25T18:18:04.438 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.435+0000 7fa081405640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa07c0a9ab0 con 0x7fa07c0a9470 2024-03-25T18:18:04.438 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.435+0000 7fa081405640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa07c0b2400 con 0x7fa07c0b4360 2024-03-25T18:18:04.438 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.435+0000 7fa081405640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa07c0aa2a0 con 0x7fa07c0b10d0 2024-03-25T18:18:04.438 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.435+0000 7fa081405640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:04.438 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.435+0000 7fa081405640 10 monclient(hunting): authenticate will time out at 7565.746156s 2024-03-25T18:18:04.438 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.435+0000 7fa07affd640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7fa07c0b10d0 0x7fa07c0b4ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:35582/0 (socket says 172.21.15.156:35582) 2024-03-25T18:18:04.438 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.435+0000 7fa07affd640 1 -- 172.21.15.156:0/3510225376 learned_addr learned my addr 172.21.15.156:0/3510225376 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:18:04.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.456+0000 7fa079ffb640 1 -- 172.21.15.156:0/3510225376 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1705052648 0 0) 0x7fa07c0b2400 con 0x7fa07c0b4360 2024-03-25T18:18:04.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.457+0000 7fa079ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:04.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.457+0000 7fa079ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:04.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.457+0000 7fa079ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:04.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.457+0000 7fa079ffb640 10 monclient(hunting): my global_id is 4666 2024-03-25T18:18:04.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.457+0000 7fa079ffb640 1 -- 172.21.15.156:0/3510225376 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa068003520 con 0x7fa07c0b4360 2024-03-25T18:18:04.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.477+0000 7fa079ffb640 1 -- 172.21.15.156:0/3510225376 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145743284 0 0) 0x7fa07c0a9ab0 con 0x7fa07c0a9470 2024-03-25T18:18:04.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.477+0000 7fa079ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:04.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.477+0000 7fa079ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:04.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.477+0000 7fa079ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:04.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.477+0000 7fa079ffb640 10 monclient(hunting): my global_id is 4662 2024-03-25T18:18:04.478 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.477+0000 7fa079ffb640 1 -- 172.21.15.156:0/3510225376 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa07c0b2400 con 0x7fa07c0a9470 2024-03-25T18:18:04.499 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.498+0000 7fa079ffb640 1 -- 172.21.15.156:0/3510225376 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4010885410 0 0) 0x7fa07c0b2400 con 0x7fa07c0a9470 2024-03-25T18:18:04.499 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.498+0000 7fa079ffb640 1 -- 172.21.15.156:0/3510225376 >> v1:172.21.15.156:6790/0 conn(0x7fa07c0b10d0 legacy=0x7fa07c0b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:04.499 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.498+0000 7fa079ffb640 1 -- 172.21.15.156:0/3510225376 >> v1:172.21.15.194:6789/0 conn(0x7fa07c0b4360 legacy=0x7fa07c0b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:04.499 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.498+0000 7fa079ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:18:04.499 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.498+0000 7fa079ffb640 1 monclient: found mon.noname-a 2024-03-25T18:18:04.499 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.498+0000 7fa079ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:04.499 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.498+0000 7fa079ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:18:04.499 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.498+0000 7fa079ffb640 1 -- 172.21.15.156:0/3510225376 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa07c0b14b0 con 0x7fa07c0a9470 2024-03-25T18:18:04.500 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.498+0000 7fa079ffb640 10 monclient: _finish_auth 0 2024-03-25T18:18:04.500 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.498+0000 7fa079ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:18:04.500 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.498+0000 7fa079ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:04.500 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.498+0000 7fa079ffb640 1 -- 172.21.15.156:0/3510225376 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fa06c001670 con 0x7fa07c0a9470 2024-03-25T18:18:04.500 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.498+0000 7fa079ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:04.500 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.498+0000 7fa079ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:04.500 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.498+0000 7fa079ffb640 10 monclient: dump: 2024-03-25T18:18:04.500 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:04.500 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:04.500 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:04.500 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:04.500 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:04.500 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:04.500 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:04.500 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:04.501 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:04.501 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:04.501 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.498+0000 7fa081405640 5 monclient: authenticate success, global_id 4662 2024-03-25T18:18:04.501 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.498+0000 7fa081405640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:04.508 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.507+0000 7fa079ffb640 1 -- 172.21.15.156:0/3510225376 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa06c0017d0 con 0x7fa07c0a9470 2024-03-25T18:18:04.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.507+0000 7fa079ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:04.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.507+0000 7fa079ffb640 1 -- 172.21.15.156:0/3510225376 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fa06c001ba0 con 0x7fa07c0a9470 2024-03-25T18:18:04.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.507+0000 7fa081405640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:04.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.507+0000 7fa081405640 4 set_mon_vals no callback set 2024-03-25T18:18:04.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.507+0000 7fa079ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:04.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.507+0000 7fa079ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:04.509 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.507+0000 7fa079ffb640 10 monclient: dump: 2024-03-25T18:18:04.509 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:04.509 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:04.509 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:04.509 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:04.510 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:04.510 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:04.510 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:04.510 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:04.510 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:04.510 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:04.513 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.508+0000 7fa081405640 10 monclient: shutdown 2024-03-25T18:18:04.513 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.508+0000 7fa081405640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:04.513 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.508+0000 7fa081405640 1 -- 172.21.15.156:0/3510225376 >> v1:172.21.15.156:6789/0 conn(0x7fa07c0a9470 legacy=0x7fa07c14dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:04.513 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.508+0000 7fa081405640 1 -- 172.21.15.156:0/3510225376 shutdown_connections 2024-03-25T18:18:04.513 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.508+0000 7fa081405640 1 -- 172.21.15.156:0/3510225376 >> 172.21.15.156:0/3510225376 conn(0x7fa07c0617a0 msgr2=0x7fa07c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:04.513 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.508+0000 7fa081405640 1 -- 172.21.15.156:0/3510225376 shutdown_connections 2024-03-25T18:18:04.513 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.508+0000 7fa081405640 1 -- 172.21.15.156:0/3510225376 wait complete. 2024-03-25T18:18:04.513 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.509+0000 7fa081405640 10 monclient: build_initial_monmap 2024-03-25T18:18:04.513 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.509+0000 7fa081405640 10 monclient: monmap: 2024-03-25T18:18:04.513 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:18:04.513 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:04.514 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:18:04.514 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:18:04.514 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:04.514 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:18:04.514 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:18:04.514 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:18:04.514 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:18:04.514 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:04.514 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.510+0000 7fa081405640 1 librados: starting msgr at 2024-03-25T18:18:04.514 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.510+0000 7fa081405640 1 librados: starting objecter 2024-03-25T18:18:04.514 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.510+0000 7fa081405640 1 Processor -- start 2024-03-25T18:18:04.514 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.510+0000 7fa081405640 1 -- start start 2024-03-25T18:18:04.514 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.510+0000 7fa081405640 1 librados: setting wanted keys 2024-03-25T18:18:04.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.510+0000 7fa081405640 1 librados: calling monclient init 2024-03-25T18:18:04.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.510+0000 7fa081405640 10 monclient: init 2024-03-25T18:18:04.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.511+0000 7fa081405640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:04.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.511+0000 7fa081405640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:04.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.511+0000 7fa081405640 10 monclient(hunting): picked mon.noname-a con 0x7fa07c0a9470 addr v1:172.21.15.156:6789/0 2024-03-25T18:18:04.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.511+0000 7fa081405640 10 monclient(hunting): picked mon.noname-b con 0x7fa07c0b10d0 addr v1:172.21.15.194:6789/0 2024-03-25T18:18:04.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.511+0000 7fa081405640 10 monclient(hunting): picked mon.noname-c con 0x7fa07c0b4360 addr v1:172.21.15.156:6790/0 2024-03-25T18:18:04.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.511+0000 7fa081405640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa07c1d7800 con 0x7fa07c0a9470 2024-03-25T18:18:04.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.511+0000 7fa081405640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa07c1d7490 con 0x7fa07c0b10d0 2024-03-25T18:18:04.515 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.511+0000 7fa081405640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa07c13f850 con 0x7fa07c0b4360 2024-03-25T18:18:04.516 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.511+0000 7fa081405640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:04.516 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.511+0000 7fa081405640 10 monclient(hunting): authenticate will time out at 7565.822333s 2024-03-25T18:18:04.516 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.512+0000 7fa07a7fc640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fa07c0b10d0 0x7fa07c139e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:52538/0 (socket says 172.21.15.156:52538) 2024-03-25T18:18:04.516 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.512+0000 7fa07a7fc640 1 -- 172.21.15.156:0/1928447006 learned_addr learned my addr 172.21.15.156:0/1928447006 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:18:04.516 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.512+0000 7fa0737fe640 10 client.?.objecter ms_handle_connect 0x7fa07c0b10d0 2024-03-25T18:18:04.516 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.512+0000 7fa0737fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:04.528 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.527+0000 7fa0737fe640 10 client.?.objecter ms_handle_connect 0x7fa07c0b4360 2024-03-25T18:18:04.528 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.527+0000 7fa0737fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:04.572 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.571+0000 7fa0737fe640 10 client.?.objecter ms_handle_connect 0x7fa07c0a9470 2024-03-25T18:18:04.572 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.571+0000 7fa0737fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:04.596 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.595+0000 7fa0737fe640 1 -- 172.21.15.156:0/1928447006 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3228018593 0 0) 0x7fa07c1d7800 con 0x7fa07c0a9470 2024-03-25T18:18:04.596 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.595+0000 7fa0737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:04.596 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.595+0000 7fa0737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:04.596 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.595+0000 7fa0737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:04.596 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.595+0000 7fa0737fe640 10 monclient(hunting): my global_id is 4665 2024-03-25T18:18:04.596 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.595+0000 7fa0737fe640 1 -- 172.21.15.156:0/1928447006 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa05c004eb0 con 0x7fa07c0a9470 2024-03-25T18:18:04.611 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.609+0000 7fa0737fe640 1 -- 172.21.15.156:0/1928447006 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1549125272 0 0) 0x7fa05c004eb0 con 0x7fa07c0a9470 2024-03-25T18:18:04.611 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.609+0000 7fa0737fe640 1 -- 172.21.15.156:0/1928447006 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa07c1d7800 con 0x7fa07c0a9470 2024-03-25T18:18:04.611 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.609+0000 7fa0737fe640 1 -- 172.21.15.156:0/1928447006 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fa074001c20 con 0x7fa07c0a9470 2024-03-25T18:18:04.611 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.609+0000 7fa0737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:04.611 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.609+0000 7fa0737fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:04.611 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.609+0000 7fa0737fe640 10 monclient(hunting): dump: 2024-03-25T18:18:04.611 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:04.611 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:04.611 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:04.611 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:04.612 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:04.612 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:04.612 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:04.612 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:04.612 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:04.612 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:04.612 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.609+0000 7fa0737fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:04.629 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.628+0000 7fa0737fe640 1 -- 172.21.15.156:0/1928447006 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1320080220 0 0) 0x7fa07c1d7800 con 0x7fa07c0a9470 2024-03-25T18:18:04.629 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.628+0000 7fa0737fe640 1 -- 172.21.15.156:0/1928447006 >> v1:172.21.15.156:6790/0 conn(0x7fa07c0b4360 legacy=0x7fa07c1e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:04.629 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.628+0000 7fa0737fe640 1 -- 172.21.15.156:0/1928447006 >> v1:172.21.15.194:6789/0 conn(0x7fa07c0b10d0 legacy=0x7fa07c139e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:04.629 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.628+0000 7fa0737fe640 10 monclient: _finish_hunting 0 2024-03-25T18:18:04.629 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.628+0000 7fa0737fe640 1 monclient: found mon.a 2024-03-25T18:18:04.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.628+0000 7fa0737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:04.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.628+0000 7fa0737fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:18:04.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.628+0000 7fa0737fe640 1 -- 172.21.15.156:0/1928447006 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa07c13a580 con 0x7fa07c0a9470 2024-03-25T18:18:04.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.628+0000 7fa0737fe640 10 monclient: _finish_auth 0 2024-03-25T18:18:04.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.628+0000 7fa0737fe640 10 monclient: _check_auth_tickets 2024-03-25T18:18:04.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.628+0000 7fa0737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:04.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.628+0000 7fa081405640 5 monclient: authenticate success, global_id 4665 2024-03-25T18:18:04.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.628+0000 7fa081405640 10 monclient: _renew_subs 2024-03-25T18:18:04.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.628+0000 7fa081405640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:18:04.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.628+0000 7fa081405640 1 -- 172.21.15.156:0/1928447006 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa07c13a7d0 con 0x7fa07c0a9470 2024-03-25T18:18:04.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.628+0000 7fa081405640 10 client.4665.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:04.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.628+0000 7fa081405640 10 monclient: _renew_subs 2024-03-25T18:18:04.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.628+0000 7fa081405640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:18:04.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.628+0000 7fa081405640 1 -- 172.21.15.156:0/1928447006 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa07c0abba0 con 0x7fa07c0a9470 2024-03-25T18:18:04.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.628+0000 7fa081405640 1 librados: init done 2024-03-25T18:18:04.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.629+0000 7fa081405640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:04.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.629+0000 7fa081405640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:04.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.629+0000 7fa081405640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:18:04.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.629+0000 7fa081405640 1 -- 172.21.15.156:0/1928447006 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa07c1d7620 con 0x7fa07c0a9470 2024-03-25T18:18:04.923 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.922+0000 7fa0737fe640 1 -- 172.21.15.156:0/1928447006 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0740020d0 con 0x7fa07c0a9470 2024-03-25T18:18:04.923 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.922+0000 7fa0737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:04.923 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.922+0000 7fa0737fe640 1 -- 172.21.15.156:0/1928447006 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fa074004d80 con 0x7fa07c0a9470 2024-03-25T18:18:04.924 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.922+0000 7fa0737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:04.924 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.922+0000 7fa0737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:04.924 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.922+0000 7fa0737fe640 10 monclient: dump: 2024-03-25T18:18:04.924 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:04.924 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:04.924 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:04.924 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:04.924 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:04.924 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:04.924 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:04.924 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:04.924 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:04.924 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:04.925 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.922+0000 7fa0797fa640 4 set_mon_vals no callback set 2024-03-25T18:18:04.925 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.923+0000 7fa0737fe640 1 -- 172.21.15.156:0/1928447006 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7fa074004fa0 con 0x7fa07c0a9470 2024-03-25T18:18:04.925 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.924+0000 7fa07a7fc640 10 monclient: get_auth_request con 0x7fa05c06dc90 auth_method 0 2024-03-25T18:18:04.925 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.924+0000 7fa0737fe640 10 client.4665.objecter ms_handle_connect 0x7fa05c06dc90 2024-03-25T18:18:04.932 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.931+0000 7fa0737fe640 1 -- 172.21.15.156:0/1928447006 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7fa074054b80 con 0x7fa07c0a9470 2024-03-25T18:18:04.932 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.931+0000 7fa0737fe640 10 client.4665.objecter ms_dispatch 0x7fa07c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:04.932 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.931+0000 7fa0737fe640 3 client.4665.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:04.932 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.931+0000 7fa0737fe640 3 client.4665.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:04.932 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.931+0000 7fa0737fe640 20 client.4665.objecter dump_active .. 0 homeless 2024-03-25T18:18:04.932 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.931+0000 7fa0737fe640 1 -- 172.21.15.156:0/1928447006 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa0740504c0 con 0x7fa07c0a9470 2024-03-25T18:18:04.933 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.931+0000 7fa0737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:04.933 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:04.931+0000 7fa0737fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:05.173 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:05.171+0000 7fa081405640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:05.173 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:05.171+0000 7fa081405640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:05.173 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:05.171+0000 7fa081405640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:18:05.173 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:05.171+0000 7fa081405640 1 -- 172.21.15.156:0/1928447006 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa07c0627b0 con 0x7fa07c0a9470 2024-03-25T18:18:05.215 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:05.213+0000 7fa0737fe640 1 -- 172.21.15.156:0/1928447006 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7fa0740525c0 con 0x7fa07c0a9470 2024-03-25T18:18:05.215 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:05.213+0000 7fa0737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:05.215 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:05.213+0000 7fa0737fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:05.215 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:18:05.215 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:05.218 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:05.216+0000 7fa081405640 10 librados: watch_flush enter 2024-03-25T18:18:05.218 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:05.216+0000 7fa081405640 10 librados: watch_flush exit 2024-03-25T18:18:05.218 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:05.216+0000 7fa081405640 20 client.4665.objecter shutdown clearing up homeless session... 2024-03-25T18:18:05.218 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:05.216+0000 7fa081405640 10 client.4665.objecter successfully canceled tick 2024-03-25T18:18:05.218 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:05.216+0000 7fa081405640 1 -- 172.21.15.156:0/1928447006 >> v1:172.21.15.194:6800/2946427713 conn(0x7fa05c06dc90 legacy=0x7fa05c070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:05.218 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:05.216+0000 7fa081405640 10 monclient: shutdown 2024-03-25T18:18:05.218 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:05.216+0000 7fa081405640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:05.218 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:05.216+0000 7fa081405640 1 -- 172.21.15.156:0/1928447006 >> v1:172.21.15.156:6789/0 conn(0x7fa07c0a9470 legacy=0x7fa07c1d7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:05.218 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:05.216+0000 7fa081405640 1 -- 172.21.15.156:0/1928447006 shutdown_connections 2024-03-25T18:18:05.218 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:05.216+0000 7fa081405640 1 -- 172.21.15.156:0/1928447006 >> 172.21.15.156:0/1928447006 conn(0x7fa07c0617a0 msgr2=0x7fa07c145e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:05.218 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:05.216+0000 7fa081405640 1 -- 172.21.15.156:0/1928447006 shutdown_connections 2024-03-25T18:18:05.219 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:05.216+0000 7fa081405640 1 -- 172.21.15.156:0/1928447006 wait complete. 2024-03-25T18:18:05.219 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:05.216+0000 7fa081405640 1 librados: shutdown 2024-03-25T18:18:05.229 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:12.035 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:12.153 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.152+0000 7f9b5004a640 10 monclient: get_monmap_and_config 2024-03-25T18:18:12.154 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.152+0000 7f9b5004a640 10 monclient: build_initial_monmap 2024-03-25T18:18:12.154 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.152+0000 7f9b5004a640 10 monclient: monmap: 2024-03-25T18:18:12.154 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:18:12.154 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:12.154 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:18:12.153858+0000 2024-03-25T18:18:12.154 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:18:12.153858+0000 2024-03-25T18:18:12.154 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:12.154 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:12.154 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:18:12.154 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:18:12.154 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:18:12.154 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:12.155 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.153+0000 7f9b5004a640 1 Processor -- start 2024-03-25T18:18:12.155 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.153+0000 7f9b5004a640 1 -- start start 2024-03-25T18:18:12.155 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.153+0000 7f9b5004a640 10 monclient: init 2024-03-25T18:18:12.156 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.154+0000 7f9b5004a640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:12.156 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.154+0000 7f9b5004a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:12.156 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.154+0000 7f9b5004a640 10 monclient(hunting): picked mon.noname-b con 0x7f9b480a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:18:12.156 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.154+0000 7f9b5004a640 10 monclient(hunting): picked mon.noname-a con 0x7f9b4813f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:18:12.156 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.154+0000 7f9b5004a640 10 monclient(hunting): picked mon.noname-c con 0x7f9b481430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:18:12.156 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.154+0000 7f9b5004a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b48142520 con 0x7f9b4813f710 2024-03-25T18:18:12.156 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.154+0000 7f9b5004a640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b480b28a0 con 0x7f9b481430c0 2024-03-25T18:18:12.156 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.154+0000 7f9b5004a640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b480b2a30 con 0x7f9b480a98f0 2024-03-25T18:18:12.156 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.154+0000 7f9b5004a640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:12.156 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.154+0000 7f9b5004a640 10 monclient(hunting): authenticate will time out at 7573.465524s 2024-03-25T18:18:12.159 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.158+0000 7f9b4e5e7640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f9b481430c0 0x7f9b4814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:35038/0 (socket says 172.21.15.156:35038) 2024-03-25T18:18:12.159 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.158+0000 7f9b4e5e7640 1 -- 172.21.15.156:0/2397735050 learned_addr learned my addr 172.21.15.156:0/2397735050 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:18:12.179 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.178+0000 7f9b4cde4640 1 -- 172.21.15.156:0/2397735050 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 602223903 0 0) 0x7f9b480b28a0 con 0x7f9b481430c0 2024-03-25T18:18:12.180 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.178+0000 7f9b4cde4640 10 monclient(hunting): _negotiate 2024-03-25T18:18:12.180 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.178+0000 7f9b4cde4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:12.180 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.178+0000 7f9b4cde4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:12.180 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.178+0000 7f9b4cde4640 10 monclient(hunting): my global_id is 4672 2024-03-25T18:18:12.180 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.178+0000 7f9b4cde4640 1 -- 172.21.15.156:0/2397735050 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b34003520 con 0x7f9b481430c0 2024-03-25T18:18:12.180 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.178+0000 7f9b4cde4640 1 -- 172.21.15.156:0/2397735050 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033084758 0 0) 0x7f9b480b2a30 con 0x7f9b480a98f0 2024-03-25T18:18:12.180 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.178+0000 7f9b4cde4640 10 monclient(hunting): _negotiate 2024-03-25T18:18:12.180 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.178+0000 7f9b4cde4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:12.180 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.178+0000 7f9b4cde4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:12.180 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.178+0000 7f9b4cde4640 10 monclient(hunting): my global_id is 4676 2024-03-25T18:18:12.180 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.178+0000 7f9b4cde4640 1 -- 172.21.15.156:0/2397735050 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b480b28a0 con 0x7f9b480a98f0 2024-03-25T18:18:12.193 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.191+0000 7f9b4cde4640 1 -- 172.21.15.156:0/2397735050 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4286388371 0 0) 0x7f9b34003520 con 0x7f9b481430c0 2024-03-25T18:18:12.193 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.191+0000 7f9b4cde4640 1 -- 172.21.15.156:0/2397735050 >> v1:172.21.15.156:6790/0 conn(0x7f9b480a98f0 legacy=0x7f9b480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:12.193 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.191+0000 7f9b4cde4640 1 -- 172.21.15.156:0/2397735050 >> v1:172.21.15.156:6789/0 conn(0x7f9b4813f710 legacy=0x7f9b481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:12.193 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.191+0000 7f9b4cde4640 10 monclient: _finish_hunting 0 2024-03-25T18:18:12.193 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.191+0000 7f9b4cde4640 1 monclient: found mon.noname-c 2024-03-25T18:18:12.193 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.191+0000 7f9b4cde4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:12.193 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.191+0000 7f9b4cde4640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:18:12.193 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.191+0000 7f9b4cde4640 1 -- 172.21.15.156:0/2397735050 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b480b44c0 con 0x7f9b481430c0 2024-03-25T18:18:12.193 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.191+0000 7f9b4cde4640 10 monclient: _finish_auth 0 2024-03-25T18:18:12.193 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.191+0000 7f9b4cde4640 10 monclient: _check_auth_tickets 2024-03-25T18:18:12.193 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.191+0000 7f9b4cde4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:12.193 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.191+0000 7f9b4cde4640 1 -- 172.21.15.156:0/2397735050 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9b40001a10 con 0x7f9b481430c0 2024-03-25T18:18:12.193 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.191+0000 7f9b4cde4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:12.193 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.191+0000 7f9b4cde4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:12.194 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.191+0000 7f9b4cde4640 10 monclient: dump: 2024-03-25T18:18:12.194 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:12.194 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:12.194 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:12.194 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:12.194 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:12.194 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:12.194 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:12.194 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:12.194 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:12.194 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:12.194 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.191+0000 7f9b5004a640 5 monclient: authenticate success, global_id 4672 2024-03-25T18:18:12.194 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.191+0000 7f9b5004a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:12.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.198+0000 7f9b4cde4640 1 -- 172.21.15.156:0/2397735050 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b40001b70 con 0x7f9b481430c0 2024-03-25T18:18:12.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.198+0000 7f9b4cde4640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:12.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.198+0000 7f9b4cde4640 1 -- 172.21.15.156:0/2397735050 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9b40001f80 con 0x7f9b481430c0 2024-03-25T18:18:12.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.198+0000 7f9b5004a640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:12.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.198+0000 7f9b5004a640 4 set_mon_vals no callback set 2024-03-25T18:18:12.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.198+0000 7f9b4cde4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:12.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.198+0000 7f9b4cde4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:12.200 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.198+0000 7f9b4cde4640 10 monclient: dump: 2024-03-25T18:18:12.200 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:12.200 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:12.200 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:12.200 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:12.200 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:12.201 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:12.201 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:12.201 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:12.201 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:12.201 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:12.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.199+0000 7f9b5004a640 10 monclient: shutdown 2024-03-25T18:18:12.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.199+0000 7f9b5004a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:12.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.199+0000 7f9b5004a640 1 -- 172.21.15.156:0/2397735050 >> v1:172.21.15.194:6789/0 conn(0x7f9b481430c0 legacy=0x7f9b4814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:12.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.199+0000 7f9b5004a640 1 -- 172.21.15.156:0/2397735050 shutdown_connections 2024-03-25T18:18:12.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.199+0000 7f9b5004a640 1 -- 172.21.15.156:0/2397735050 >> 172.21.15.156:0/2397735050 conn(0x7f9b480b7ea0 msgr2=0x7f9b480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:12.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.200+0000 7f9b5004a640 1 -- 172.21.15.156:0/2397735050 shutdown_connections 2024-03-25T18:18:12.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.200+0000 7f9b5004a640 1 -- 172.21.15.156:0/2397735050 wait complete. 2024-03-25T18:18:12.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.200+0000 7f9b5004a640 10 monclient: build_initial_monmap 2024-03-25T18:18:12.203 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.200+0000 7f9b5004a640 10 monclient: monmap: 2024-03-25T18:18:12.203 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:18:12.203 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:12.204 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:18:12.204 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:18:12.204 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:12.204 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:18:12.204 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:18:12.204 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:18:12.204 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:18:12.204 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:12.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.200+0000 7f9b5004a640 1 librados: starting msgr at 2024-03-25T18:18:12.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.200+0000 7f9b5004a640 1 librados: starting objecter 2024-03-25T18:18:12.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.201+0000 7f9b5004a640 1 Processor -- start 2024-03-25T18:18:12.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.201+0000 7f9b5004a640 1 -- start start 2024-03-25T18:18:12.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.201+0000 7f9b5004a640 1 librados: setting wanted keys 2024-03-25T18:18:12.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.201+0000 7f9b5004a640 1 librados: calling monclient init 2024-03-25T18:18:12.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.201+0000 7f9b5004a640 10 monclient: init 2024-03-25T18:18:12.204 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.201+0000 7f9b5004a640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:12.205 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.201+0000 7f9b5004a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:12.205 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.201+0000 7f9b5004a640 10 monclient(hunting): picked mon.noname-c con 0x7f9b480a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:18:12.205 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.201+0000 7f9b5004a640 10 monclient(hunting): picked mon.noname-a con 0x7f9b4813f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:18:12.205 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.201+0000 7f9b5004a640 10 monclient(hunting): picked mon.noname-b con 0x7f9b481430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:18:12.205 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.201+0000 7f9b5004a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b48137c60 con 0x7f9b4813f710 2024-03-25T18:18:12.205 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.201+0000 7f9b5004a640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b48139160 con 0x7f9b481430c0 2024-03-25T18:18:12.205 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.201+0000 7f9b5004a640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b481380f0 con 0x7f9b480a98f0 2024-03-25T18:18:12.205 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.201+0000 7f9b5004a640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:12.205 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.201+0000 7f9b5004a640 10 monclient(hunting): authenticate will time out at 7573.512550s 2024-03-25T18:18:12.216 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.215+0000 7f9b4dde6640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f9b480a98f0 0x7f9b481dab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:44504/0 (socket says 172.21.15.156:44504) 2024-03-25T18:18:12.216 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.215+0000 7f9b4dde6640 1 -- 172.21.15.156:0/3135201947 learned_addr learned my addr 172.21.15.156:0/3135201947 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:18:12.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.217+0000 7f9b3e7fc640 10 client.?.objecter ms_handle_connect 0x7f9b480a98f0 2024-03-25T18:18:12.217 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.217+0000 7f9b3e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:12.223 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.222+0000 7f9b3e7fc640 10 client.?.objecter ms_handle_connect 0x7f9b4813f710 2024-03-25T18:18:12.223 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.222+0000 7f9b3e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:12.225 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.224+0000 7f9b3e7fc640 10 client.?.objecter ms_handle_connect 0x7f9b481430c0 2024-03-25T18:18:12.225 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.224+0000 7f9b3e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:12.226 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.225+0000 7f9b3e7fc640 1 -- 172.21.15.156:0/3135201947 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1583760314 0 0) 0x7f9b481380f0 con 0x7f9b480a98f0 2024-03-25T18:18:12.226 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.225+0000 7f9b3e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:12.226 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.225+0000 7f9b3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:12.226 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.225+0000 7f9b3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:12.227 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.225+0000 7f9b3e7fc640 10 monclient(hunting): my global_id is 4679 2024-03-25T18:18:12.227 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.225+0000 7f9b3e7fc640 1 -- 172.21.15.156:0/3135201947 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b2c004eb0 con 0x7f9b480a98f0 2024-03-25T18:18:12.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.244+0000 7f9b3e7fc640 1 -- 172.21.15.156:0/3135201947 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1608739065 0 0) 0x7f9b48139160 con 0x7f9b481430c0 2024-03-25T18:18:12.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.244+0000 7f9b3e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:12.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.244+0000 7f9b3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:12.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.244+0000 7f9b3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:12.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.244+0000 7f9b3e7fc640 10 monclient(hunting): my global_id is 4675 2024-03-25T18:18:12.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.244+0000 7f9b3e7fc640 1 -- 172.21.15.156:0/3135201947 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b481380f0 con 0x7f9b481430c0 2024-03-25T18:18:12.248 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.246+0000 7f9b3e7fc640 1 -- 172.21.15.156:0/3135201947 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3598546180 0 0) 0x7f9b48137c60 con 0x7f9b4813f710 2024-03-25T18:18:12.248 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.247+0000 7f9b3e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:12.248 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.247+0000 7f9b3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:12.248 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.247+0000 7f9b3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:12.248 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.247+0000 7f9b3e7fc640 10 monclient(hunting): my global_id is 4671 2024-03-25T18:18:12.248 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.247+0000 7f9b3e7fc640 1 -- 172.21.15.156:0/3135201947 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b48139160 con 0x7f9b4813f710 2024-03-25T18:18:12.260 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.259+0000 7f9b3e7fc640 1 -- 172.21.15.156:0/3135201947 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 951355597 0 0) 0x7f9b2c004eb0 con 0x7f9b480a98f0 2024-03-25T18:18:12.261 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.259+0000 7f9b3e7fc640 1 -- 172.21.15.156:0/3135201947 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b48137c60 con 0x7f9b480a98f0 2024-03-25T18:18:12.261 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.259+0000 7f9b3e7fc640 1 -- 172.21.15.156:0/3135201947 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9b44001a10 con 0x7f9b480a98f0 2024-03-25T18:18:12.261 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.259+0000 7f9b3e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:12.261 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.259+0000 7f9b3e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:12.261 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.259+0000 7f9b3e7fc640 10 monclient(hunting): dump: 2024-03-25T18:18:12.261 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:12.261 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:12.261 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:12.261 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:12.261 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:12.261 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:12.261 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:12.261 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:12.261 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:12.261 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:12.261 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.259+0000 7f9b3e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:12.340 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.339+0000 7f9b3e7fc640 1 -- 172.21.15.156:0/3135201947 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4007717291 0 0) 0x7f9b48139160 con 0x7f9b4813f710 2024-03-25T18:18:12.340 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.339+0000 7f9b3e7fc640 1 -- 172.21.15.156:0/3135201947 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b2c004eb0 con 0x7f9b4813f710 2024-03-25T18:18:12.340 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.339+0000 7f9b3e7fc640 1 -- 172.21.15.156:0/3135201947 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9b38001860 con 0x7f9b4813f710 2024-03-25T18:18:12.340 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.339+0000 7f9b3e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:12.340 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.339+0000 7f9b3e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:12.340 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.339+0000 7f9b3e7fc640 10 monclient(hunting): dump: 2024-03-25T18:18:12.340 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:12.340 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:12.340 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:12.340 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:12.341 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:12.341 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:12.341 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:12.341 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:12.341 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:12.341 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:12.363 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.361+0000 7f9b3e7fc640 1 -- 172.21.15.156:0/3135201947 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2977000487 0 0) 0x7f9b2c004eb0 con 0x7f9b4813f710 2024-03-25T18:18:12.363 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.361+0000 7f9b3e7fc640 1 -- 172.21.15.156:0/3135201947 >> v1:172.21.15.156:6790/0 conn(0x7f9b480a98f0 legacy=0x7f9b481dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:12.363 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.362+0000 7f9b3e7fc640 1 -- 172.21.15.156:0/3135201947 >> v1:172.21.15.194:6789/0 conn(0x7f9b481430c0 legacy=0x7f9b481e2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:12.363 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.362+0000 7f9b3e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:18:12.363 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.362+0000 7f9b3e7fc640 1 monclient: found mon.a 2024-03-25T18:18:12.363 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.362+0000 7f9b3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:12.363 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.362+0000 7f9b3e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:18:12.363 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.362+0000 7f9b3e7fc640 1 -- 172.21.15.156:0/3135201947 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b481397f0 con 0x7f9b4813f710 2024-03-25T18:18:12.363 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.362+0000 7f9b3e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:18:12.364 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.362+0000 7f9b3e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:18:12.364 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.362+0000 7f9b3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:12.364 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.362+0000 7f9b5004a640 5 monclient: authenticate success, global_id 4671 2024-03-25T18:18:12.364 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.362+0000 7f9b5004a640 10 monclient: _renew_subs 2024-03-25T18:18:12.364 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.362+0000 7f9b5004a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:18:12.364 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.362+0000 7f9b5004a640 1 -- 172.21.15.156:0/3135201947 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b481db280 con 0x7f9b4813f710 2024-03-25T18:18:12.364 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.362+0000 7f9b5004a640 10 client.4671.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:12.364 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.362+0000 7f9b5004a640 10 monclient: _renew_subs 2024-03-25T18:18:12.364 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.362+0000 7f9b5004a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:18:12.364 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.362+0000 7f9b5004a640 1 -- 172.21.15.156:0/3135201947 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b48133960 con 0x7f9b4813f710 2024-03-25T18:18:12.364 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.362+0000 7f9b5004a640 1 librados: init done 2024-03-25T18:18:12.365 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.363+0000 7f9b5004a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:12.365 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.363+0000 7f9b5004a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:12.365 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.363+0000 7f9b5004a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:18:12.365 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.363+0000 7f9b5004a640 1 -- 172.21.15.156:0/3135201947 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b48139a40 con 0x7f9b4813f710 2024-03-25T18:18:12.675 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.673+0000 7f9b3e7fc640 1 -- 172.21.15.156:0/3135201947 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b38001d30 con 0x7f9b4813f710 2024-03-25T18:18:12.675 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.673+0000 7f9b3e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:12.675 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.673+0000 7f9b3e7fc640 1 -- 172.21.15.156:0/3135201947 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9b38003790 con 0x7f9b4813f710 2024-03-25T18:18:12.675 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.673+0000 7f9b3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:12.675 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.673+0000 7f9b3e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:12.675 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.673+0000 7f9b3e7fc640 10 monclient: dump: 2024-03-25T18:18:12.675 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:12.675 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:12.675 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:12.675 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:12.676 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:12.676 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:12.676 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:12.676 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:12.676 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:12.676 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:12.676 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.673+0000 7f9b3ffff640 4 set_mon_vals no callback set 2024-03-25T18:18:12.676 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.674+0000 7f9b3e7fc640 1 -- 172.21.15.156:0/3135201947 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f9b380197e0 con 0x7f9b4813f710 2024-03-25T18:18:12.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.675+0000 7f9b4dde6640 10 monclient: get_auth_request con 0x7f9b2c06dfb0 auth_method 0 2024-03-25T18:18:12.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.675+0000 7f9b3e7fc640 10 client.4671.objecter ms_handle_connect 0x7f9b2c06dfb0 2024-03-25T18:18:12.681 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.680+0000 7f9b3e7fc640 1 -- 172.21.15.156:0/3135201947 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f9b38053520 con 0x7f9b4813f710 2024-03-25T18:18:12.681 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.680+0000 7f9b3e7fc640 10 client.4671.objecter ms_dispatch 0x7f9b48000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:12.681 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.680+0000 7f9b3e7fc640 3 client.4671.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:12.681 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.680+0000 7f9b3e7fc640 3 client.4671.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:12.682 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.680+0000 7f9b3e7fc640 20 client.4671.objecter dump_active .. 0 homeless 2024-03-25T18:18:12.682 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.680+0000 7f9b3e7fc640 1 -- 172.21.15.156:0/3135201947 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9b3804ee60 con 0x7f9b4813f710 2024-03-25T18:18:12.682 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.680+0000 7f9b3e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:12.682 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.680+0000 7f9b3e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:12.927 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.926+0000 7f9b5004a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:12.928 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.926+0000 7f9b5004a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:12.928 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.926+0000 7f9b5004a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:18:12.928 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.926+0000 7f9b5004a640 1 -- 172.21.15.156:0/3135201947 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b480627b0 con 0x7f9b4813f710 2024-03-25T18:18:12.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.960+0000 7f9b3e7fc640 1 -- 172.21.15.156:0/3135201947 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3075030351) 0x7f9b38050f60 con 0x7f9b4813f710 2024-03-25T18:18:12.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.960+0000 7f9b3e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:12.961 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.960+0000 7f9b3e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:12.961 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:18:12.961 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":10,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:08:21.657340+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:21.657340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:12.964 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.962+0000 7f9b5004a640 10 librados: watch_flush enter 2024-03-25T18:18:12.964 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.962+0000 7f9b5004a640 10 librados: watch_flush exit 2024-03-25T18:18:12.964 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.962+0000 7f9b5004a640 20 client.4671.objecter shutdown clearing up homeless session... 2024-03-25T18:18:12.965 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.962+0000 7f9b5004a640 10 client.4671.objecter successfully canceled tick 2024-03-25T18:18:12.965 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.963+0000 7f9b5004a640 1 -- 172.21.15.156:0/3135201947 >> v1:172.21.15.194:6800/2946427713 conn(0x7f9b2c06dfb0 legacy=0x7f9b2c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:12.965 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.963+0000 7f9b5004a640 10 monclient: shutdown 2024-03-25T18:18:12.965 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.963+0000 7f9b5004a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:12.965 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.963+0000 7f9b5004a640 1 -- 172.21.15.156:0/3135201947 >> v1:172.21.15.156:6789/0 conn(0x7f9b4813f710 legacy=0x7f9b48133070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:12.965 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.963+0000 7f9b5004a640 1 -- 172.21.15.156:0/3135201947 shutdown_connections 2024-03-25T18:18:12.965 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.963+0000 7f9b5004a640 1 -- 172.21.15.156:0/3135201947 >> 172.21.15.156:0/3135201947 conn(0x7f9b480b7ea0 msgr2=0x7f9b4813d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:12.965 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.963+0000 7f9b5004a640 1 -- 172.21.15.156:0/3135201947 shutdown_connections 2024-03-25T18:18:12.965 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.963+0000 7f9b5004a640 1 -- 172.21.15.156:0/3135201947 wait complete. 2024-03-25T18:18:12.965 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:12.963+0000 7f9b5004a640 1 librados: shutdown 2024-03-25T18:18:12.975 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:19.782 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:19.900 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.898+0000 7f53379cf640 10 monclient: get_monmap_and_config 2024-03-25T18:18:19.900 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.898+0000 7f53379cf640 10 monclient: build_initial_monmap 2024-03-25T18:18:19.900 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.899+0000 7f53379cf640 10 monclient: monmap: 2024-03-25T18:18:19.900 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:18:19.900 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:19.900 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:18:19.899986+0000 2024-03-25T18:18:19.900 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:18:19.899986+0000 2024-03-25T18:18:19.900 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:19.900 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:19.901 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:18:19.901 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:18:19.901 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:18:19.901 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:19.902 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.899+0000 7f53379cf640 1 Processor -- start 2024-03-25T18:18:19.902 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.900+0000 7f53379cf640 1 -- start start 2024-03-25T18:18:19.902 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.900+0000 7f53379cf640 10 monclient: init 2024-03-25T18:18:19.902 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.900+0000 7f53379cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:19.902 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.900+0000 7f53379cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:19.902 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.900+0000 7f53379cf640 10 monclient(hunting): picked mon.noname-b con 0x7f53300a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:18:19.903 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.900+0000 7f53379cf640 10 monclient(hunting): picked mon.noname-a con 0x7f533013f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:18:19.903 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.900+0000 7f53379cf640 10 monclient(hunting): picked mon.noname-c con 0x7f53301430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:18:19.903 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.900+0000 7f53379cf640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5330142520 con 0x7f533013f710 2024-03-25T18:18:19.903 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.900+0000 7f53379cf640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53300b28a0 con 0x7f53301430c0 2024-03-25T18:18:19.903 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.900+0000 7f53379cf640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53300b2a30 con 0x7f53300a98f0 2024-03-25T18:18:19.903 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.900+0000 7f53379cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:19.903 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.900+0000 7f53379cf640 10 monclient(hunting): authenticate will time out at 7581.211637s 2024-03-25T18:18:19.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.905+0000 7f5335f6c640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f53301430c0 0x7f533014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:39490/0 (socket says 172.21.15.156:39490) 2024-03-25T18:18:19.906 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.905+0000 7f5335f6c640 1 -- 172.21.15.156:0/4156144955 learned_addr learned my addr 172.21.15.156:0/4156144955 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:18:19.922 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.921+0000 7f5327fff640 1 -- 172.21.15.156:0/4156144955 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 972131867 0 0) 0x7f5330142520 con 0x7f533013f710 2024-03-25T18:18:19.922 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.921+0000 7f5327fff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:19.922 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.921+0000 7f5327fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:19.922 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.921+0000 7f5327fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:19.922 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.921+0000 7f5327fff640 10 monclient(hunting): my global_id is 4674 2024-03-25T18:18:19.922 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.921+0000 7f5327fff640 1 -- 172.21.15.156:0/4156144955 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f531c003520 con 0x7f533013f710 2024-03-25T18:18:19.927 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.926+0000 7f5327fff640 1 -- 172.21.15.156:0/4156144955 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845173648 0 0) 0x7f53300b28a0 con 0x7f53301430c0 2024-03-25T18:18:19.927 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.926+0000 7f5327fff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:19.927 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.926+0000 7f5327fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:19.927 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.926+0000 7f5327fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:19.927 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.926+0000 7f5327fff640 10 monclient(hunting): my global_id is 4678 2024-03-25T18:18:19.927 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.926+0000 7f5327fff640 1 -- 172.21.15.156:0/4156144955 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5330142520 con 0x7f53301430c0 2024-03-25T18:18:19.938 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.937+0000 7f5327fff640 1 -- 172.21.15.156:0/4156144955 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3467661236 0 0) 0x7f5330142520 con 0x7f53301430c0 2024-03-25T18:18:19.938 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.937+0000 7f5327fff640 1 -- 172.21.15.156:0/4156144955 >> v1:172.21.15.156:6790/0 conn(0x7f53300a98f0 legacy=0x7f53300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:19.938 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.937+0000 7f5327fff640 1 -- 172.21.15.156:0/4156144955 >> v1:172.21.15.156:6789/0 conn(0x7f533013f710 legacy=0x7f53301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:19.939 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.937+0000 7f5327fff640 10 monclient: _finish_hunting 0 2024-03-25T18:18:19.939 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.937+0000 7f5327fff640 1 monclient: found mon.noname-c 2024-03-25T18:18:19.939 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.937+0000 7f5327fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:19.939 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.937+0000 7f5327fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:18:19.939 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.937+0000 7f5327fff640 1 -- 172.21.15.156:0/4156144955 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53300b44c0 con 0x7f53301430c0 2024-03-25T18:18:19.939 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.937+0000 7f5327fff640 10 monclient: _finish_auth 0 2024-03-25T18:18:19.939 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.937+0000 7f5327fff640 10 monclient: _check_auth_tickets 2024-03-25T18:18:19.939 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.937+0000 7f5327fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:19.939 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.937+0000 7f5327fff640 1 -- 172.21.15.156:0/4156144955 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f53280019f0 con 0x7f53301430c0 2024-03-25T18:18:19.939 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.937+0000 7f53379cf640 5 monclient: authenticate success, global_id 4678 2024-03-25T18:18:19.939 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.937+0000 7f53379cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:19.939 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.937+0000 7f5327fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:19.939 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.937+0000 7f5327fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:19.939 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.937+0000 7f5327fff640 10 monclient: dump: 2024-03-25T18:18:19.940 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:19.940 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:19.940 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:19.940 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:19.940 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:19.940 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:19.940 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:19.940 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:19.940 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:19.940 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:19.940 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.937+0000 7f53379cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:19.952 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.952+0000 7f5327fff640 1 -- 172.21.15.156:0/4156144955 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5328001b50 con 0x7f53301430c0 2024-03-25T18:18:19.953 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.952+0000 7f5327fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:19.953 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.952+0000 7f5327fff640 1 -- 172.21.15.156:0/4156144955 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5328001f20 con 0x7f53301430c0 2024-03-25T18:18:19.953 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.952+0000 7f53379cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:19.953 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.952+0000 7f53379cf640 4 set_mon_vals no callback set 2024-03-25T18:18:19.953 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.952+0000 7f5327fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:19.953 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.952+0000 7f5327fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:19.953 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.952+0000 7f5327fff640 10 monclient: dump: 2024-03-25T18:18:19.953 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:19.953 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:19.953 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:19.953 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:19.953 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:19.953 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:19.954 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:19.954 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:19.954 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:19.954 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:19.956 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.952+0000 7f53379cf640 10 monclient: shutdown 2024-03-25T18:18:19.956 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.952+0000 7f53379cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:19.956 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.952+0000 7f53379cf640 1 -- 172.21.15.156:0/4156144955 >> v1:172.21.15.194:6789/0 conn(0x7f53301430c0 legacy=0x7f533014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:19.956 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.953+0000 7f53379cf640 1 -- 172.21.15.156:0/4156144955 shutdown_connections 2024-03-25T18:18:19.956 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.953+0000 7f53379cf640 1 -- 172.21.15.156:0/4156144955 >> 172.21.15.156:0/4156144955 conn(0x7f53300b7ea0 msgr2=0x7f53300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:19.956 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.953+0000 7f53379cf640 1 -- 172.21.15.156:0/4156144955 shutdown_connections 2024-03-25T18:18:19.956 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.953+0000 7f53379cf640 1 -- 172.21.15.156:0/4156144955 wait complete. 2024-03-25T18:18:19.956 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.953+0000 7f53379cf640 10 monclient: build_initial_monmap 2024-03-25T18:18:19.956 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.953+0000 7f53379cf640 10 monclient: monmap: 2024-03-25T18:18:19.956 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:18:19.956 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:19.957 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:18:19.957 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:18:19.957 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:19.957 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:18:19.957 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:18:19.957 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:18:19.957 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:18:19.957 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:19.957 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.954+0000 7f53379cf640 1 librados: starting msgr at 2024-03-25T18:18:19.957 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.954+0000 7f53379cf640 1 librados: starting objecter 2024-03-25T18:18:19.957 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.954+0000 7f53379cf640 1 Processor -- start 2024-03-25T18:18:19.957 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.954+0000 7f53379cf640 1 -- start start 2024-03-25T18:18:19.957 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.954+0000 7f53379cf640 1 librados: setting wanted keys 2024-03-25T18:18:19.957 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.954+0000 7f53379cf640 1 librados: calling monclient init 2024-03-25T18:18:19.957 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.954+0000 7f53379cf640 10 monclient: init 2024-03-25T18:18:19.957 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.954+0000 7f53379cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:19.958 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.954+0000 7f53379cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:19.958 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.954+0000 7f53379cf640 10 monclient(hunting): picked mon.noname-c con 0x7f53300a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:18:19.958 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.954+0000 7f53379cf640 10 monclient(hunting): picked mon.noname-a con 0x7f533013f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:18:19.958 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.954+0000 7f53379cf640 10 monclient(hunting): picked mon.noname-b con 0x7f53301430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:18:19.958 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.954+0000 7f53379cf640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53301d73f0 con 0x7f533013f710 2024-03-25T18:18:19.958 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.954+0000 7f53379cf640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53301395e0 con 0x7f53301430c0 2024-03-25T18:18:19.958 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.954+0000 7f53379cf640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53301398c0 con 0x7f53300a98f0 2024-03-25T18:18:19.958 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.954+0000 7f53379cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:19.958 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.954+0000 7f53379cf640 10 monclient(hunting): authenticate will time out at 7581.265653s 2024-03-25T18:18:19.969 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.968+0000 7f5335f6c640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f53301430c0 0x7f5330137fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:39530/0 (socket says 172.21.15.156:39530) 2024-03-25T18:18:19.969 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.968+0000 7f5335f6c640 1 -- 172.21.15.156:0/3173888579 learned_addr learned my addr 172.21.15.156:0/3173888579 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:18:19.970 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.969+0000 7f5325ffb640 10 client.?.objecter ms_handle_connect 0x7f53301430c0 2024-03-25T18:18:19.970 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.969+0000 7f5325ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:19.972 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.971+0000 7f5335f6c640 1 -- 172.21.15.156:0/3173888579 >> v1:172.21.15.194:6789/0 conn(0x7f53301430c0 legacy=0x7f5330137fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T18:18:19.972 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.971+0000 7f5335f6c640 1 -- 172.21.15.156:0/3173888579 >> v1:172.21.15.194:6789/0 conn(0x7f53301430c0 legacy=0x7f5330137fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:18:19.973 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.971+0000 7f5335f6c640 1 --1- 172.21.15.156:0/3173888579 >> v1:172.21.15.194:6789/0 conn(0x7f53301430c0 0x7f5330137fc0 :-1 s=OPENED pgs=75785 cs=1 l=1).handle_message read tag failed 2024-03-25T18:18:19.973 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.971+0000 7f5335f6c640 1 --1- 172.21.15.156:0/3173888579 >> v1:172.21.15.194:6789/0 conn(0x7f53301430c0 0x7f5330137fc0 :-1 s=OPENED pgs=75785 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:18:19.973 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.971+0000 7f5325ffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.194:6789/0 2024-03-25T18:18:19.974 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.973+0000 7f5325ffb640 10 client.?.objecter ms_handle_connect 0x7f53300a98f0 2024-03-25T18:18:19.974 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.973+0000 7f5325ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:19.988 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.987+0000 7f5325ffb640 10 client.?.objecter ms_handle_connect 0x7f533013f710 2024-03-25T18:18:19.988 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.987+0000 7f5325ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:19.993 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.992+0000 7f5325ffb640 1 -- 172.21.15.156:0/3173888579 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1568001048 0 0) 0x7f53301398c0 con 0x7f53300a98f0 2024-03-25T18:18:19.993 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.992+0000 7f5325ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:19.993 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.992+0000 7f5325ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:19.993 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.992+0000 7f5325ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:19.993 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.992+0000 7f5325ffb640 10 monclient(hunting): my global_id is 4685 2024-03-25T18:18:19.993 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:19.992+0000 7f5325ffb640 1 -- 172.21.15.156:0/3173888579 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5314004eb0 con 0x7f53300a98f0 2024-03-25T18:18:20.006 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.005+0000 7f5325ffb640 1 -- 172.21.15.156:0/3173888579 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285401148 0 0) 0x7f53301d73f0 con 0x7f533013f710 2024-03-25T18:18:20.006 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.005+0000 7f5325ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:20.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.005+0000 7f5325ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:20.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.005+0000 7f5325ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:20.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.005+0000 7f5325ffb640 10 monclient(hunting): my global_id is 4677 2024-03-25T18:18:20.007 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.005+0000 7f5325ffb640 1 -- 172.21.15.156:0/3173888579 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53301398c0 con 0x7f533013f710 2024-03-25T18:18:20.013 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.012+0000 7f5325ffb640 1 -- 172.21.15.156:0/3173888579 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 540314952 0 0) 0x7f5314004eb0 con 0x7f53300a98f0 2024-03-25T18:18:20.013 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.012+0000 7f5325ffb640 1 -- 172.21.15.156:0/3173888579 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53301d73f0 con 0x7f53300a98f0 2024-03-25T18:18:20.013 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.012+0000 7f5325ffb640 1 -- 172.21.15.156:0/3173888579 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f53200018c0 con 0x7f53300a98f0 2024-03-25T18:18:20.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.012+0000 7f5325ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:20.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.012+0000 7f5325ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:20.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.012+0000 7f5325ffb640 10 monclient(hunting): dump: 2024-03-25T18:18:20.014 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:20.014 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:20.014 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:20.014 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:20.014 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:20.014 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:20.014 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:20.014 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:20.014 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:20.014 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:20.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.012+0000 7f5325ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:20.025 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.023+0000 7f5325ffb640 1 -- 172.21.15.156:0/3173888579 <== mon.2 v1:172.21.15.156:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3932686253 0 0) 0x7f53301d73f0 con 0x7f53300a98f0 2024-03-25T18:18:20.025 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.023+0000 7f5325ffb640 1 -- 172.21.15.156:0/3173888579 >> v1:172.21.15.194:6789/0 conn(0x7f53301430c0 legacy=0x7f5330137fc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:18:20.025 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.023+0000 7f5325ffb640 1 -- 172.21.15.156:0/3173888579 >> v1:172.21.15.156:6789/0 conn(0x7f533013f710 legacy=0x7f53301d4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:20.025 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.023+0000 7f5325ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:18:20.025 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.023+0000 7f5325ffb640 1 monclient: found mon.c 2024-03-25T18:18:20.025 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.023+0000 7f5325ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:20.025 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.023+0000 7f5325ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:18:20.025 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.023+0000 7f5325ffb640 1 -- 172.21.15.156:0/3173888579 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5330137b40 con 0x7f53300a98f0 2024-03-25T18:18:20.025 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.023+0000 7f5325ffb640 10 monclient: _finish_auth 0 2024-03-25T18:18:20.025 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.023+0000 7f5325ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:18:20.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.023+0000 7f5325ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:20.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.023+0000 7f53379cf640 5 monclient: authenticate success, global_id 4685 2024-03-25T18:18:20.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.023+0000 7f53379cf640 10 monclient: _renew_subs 2024-03-25T18:18:20.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.023+0000 7f53379cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:18:20.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.023+0000 7f53379cf640 1 -- 172.21.15.156:0/3173888579 --> v1:172.21.15.156:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53301d7f90 con 0x7f53300a98f0 2024-03-25T18:18:20.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.024+0000 7f53379cf640 10 client.4685.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:20.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.024+0000 7f53379cf640 10 monclient: _renew_subs 2024-03-25T18:18:20.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.024+0000 7f53379cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:18:20.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.024+0000 7f53379cf640 1 -- 172.21.15.156:0/3173888579 --> v1:172.21.15.156:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53301d4b60 con 0x7f53300a98f0 2024-03-25T18:18:20.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.024+0000 7f53379cf640 1 librados: init done 2024-03-25T18:18:20.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.025+0000 7f53379cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:20.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.025+0000 7f53379cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:20.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.025+0000 7f53379cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:18:20.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.025+0000 7f53379cf640 1 -- 172.21.15.156:0/3173888579 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53300627b0 con 0x7f53300a98f0 2024-03-25T18:18:20.337 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.335+0000 7f5325ffb640 1 -- 172.21.15.156:0/3173888579 <== mon.2 v1:172.21.15.156:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53200035f0 con 0x7f53300a98f0 2024-03-25T18:18:20.337 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.335+0000 7f5325ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:20.337 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.335+0000 7f5325ffb640 1 -- 172.21.15.156:0/3173888579 <== mon.2 v1:172.21.15.156:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5320003a00 con 0x7f53300a98f0 2024-03-25T18:18:20.337 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.335+0000 7f5325ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:20.337 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.336+0000 7f5325ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:20.337 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.336+0000 7f5325ffb640 10 monclient: dump: 2024-03-25T18:18:20.338 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:20.338 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:20.338 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:20.338 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:20.338 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:20.338 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:20.338 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:20.338 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:20.338 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:20.338 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:20.338 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.335+0000 7f53277fe640 4 set_mon_vals no callback set 2024-03-25T18:18:20.339 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.336+0000 7f5325ffb640 1 -- 172.21.15.156:0/3173888579 <== mon.2 v1:172.21.15.156:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f5320003c20 con 0x7f53300a98f0 2024-03-25T18:18:20.339 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.337+0000 7f5334f6a640 10 monclient: get_auth_request con 0x7f531406dd90 auth_method 0 2024-03-25T18:18:20.339 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.338+0000 7f5325ffb640 10 client.4685.objecter ms_handle_connect 0x7f531406dd90 2024-03-25T18:18:20.345 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.344+0000 7f5325ffb640 1 -- 172.21.15.156:0/3173888579 <== mon.2 v1:172.21.15.156:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1740412633 0 0) 0x7f5320053600 con 0x7f53300a98f0 2024-03-25T18:18:20.346 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.344+0000 7f5325ffb640 10 client.4685.objecter ms_dispatch 0x7f5330000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:20.346 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.344+0000 7f5325ffb640 3 client.4685.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:20.346 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.344+0000 7f5325ffb640 3 client.4685.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:20.346 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.344+0000 7f5325ffb640 20 client.4685.objecter dump_active .. 0 homeless 2024-03-25T18:18:20.346 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.344+0000 7f5325ffb640 1 -- 172.21.15.156:0/3173888579 <== mon.2 v1:172.21.15.156:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f532004f110 con 0x7f53300a98f0 2024-03-25T18:18:20.346 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.344+0000 7f5325ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:20.346 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.344+0000 7f5325ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:20.563 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:10:26.553 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:20.569 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:10:26.559 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:20.574 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:10:26.541 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:20.575 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:10:26.565 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:20.577 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:10:26.544 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:20.614 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.613+0000 7f53379cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:20.614 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.613+0000 7f53379cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:20.614 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.613+0000 7f53379cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:18:20.614 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.613+0000 7f53379cf640 1 -- 172.21.15.156:0/3173888579 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53301419e0 con 0x7f53300a98f0 2024-03-25T18:18:20.619 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:10:26.610 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:20.635 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:10:26.602 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:20.641 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:10:26.631 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:20.668 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:10:26.658 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:20.768 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.767+0000 7f5325ffb640 1 -- 172.21.15.156:0/3173888579 <== mon.2 v1:172.21.15.156:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1441834839) 0x7f5320051040 con 0x7f53300a98f0 2024-03-25T18:18:20.769 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.767+0000 7f5325ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:20.769 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.767+0000 7f5325ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:20.769 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:18:20.769 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":11,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:18:20.029906+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:18:20.029906+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:20.771 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.769+0000 7f53379cf640 10 librados: watch_flush enter 2024-03-25T18:18:20.771 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.770+0000 7f53379cf640 10 librados: watch_flush exit 2024-03-25T18:18:20.772 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.770+0000 7f53379cf640 20 client.4685.objecter shutdown clearing up homeless session... 2024-03-25T18:18:20.772 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.770+0000 7f53379cf640 10 client.4685.objecter successfully canceled tick 2024-03-25T18:18:20.772 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.770+0000 7f53379cf640 1 -- 172.21.15.156:0/3173888579 >> v1:172.21.15.194:6800/2946427713 conn(0x7f531406dd90 legacy=0x7f5314070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:20.772 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.770+0000 7f53379cf640 10 monclient: shutdown 2024-03-25T18:18:20.772 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.770+0000 7f53379cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:20.772 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.770+0000 7f53379cf640 1 -- 172.21.15.156:0/3173888579 >> v1:172.21.15.156:6790/0 conn(0x7f53300a98f0 legacy=0x7f53301d7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:20.772 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.770+0000 7f53379cf640 1 -- 172.21.15.156:0/3173888579 shutdown_connections 2024-03-25T18:18:20.772 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.770+0000 7f53379cf640 1 -- 172.21.15.156:0/3173888579 >> 172.21.15.156:0/3173888579 conn(0x7f53300b7ea0 msgr2=0x7f5330074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:20.772 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.770+0000 7f53379cf640 1 -- 172.21.15.156:0/3173888579 shutdown_connections 2024-03-25T18:18:20.772 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.770+0000 7f53379cf640 1 -- 172.21.15.156:0/3173888579 wait complete. 2024-03-25T18:18:20.772 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:20.770+0000 7f53379cf640 1 librados: shutdown 2024-03-25T18:18:20.785 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:27.592 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:27.707 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.705+0000 7fc38cdf6640 10 monclient: get_monmap_and_config 2024-03-25T18:18:27.707 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.705+0000 7fc38cdf6640 10 monclient: build_initial_monmap 2024-03-25T18:18:27.707 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.705+0000 7fc38cdf6640 10 monclient: monmap: 2024-03-25T18:18:27.707 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:18:27.707 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:27.707 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:18:27.707037+0000 2024-03-25T18:18:27.707 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:18:27.707037+0000 2024-03-25T18:18:27.707 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:27.707 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:27.708 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:18:27.708 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:18:27.708 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:18:27.708 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:27.709 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.705+0000 7fc38cdf6640 1 Processor -- start 2024-03-25T18:18:27.709 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.706+0000 7fc38cdf6640 1 -- start start 2024-03-25T18:18:27.709 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.706+0000 7fc38cdf6640 10 monclient: init 2024-03-25T18:18:27.709 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.707+0000 7fc38cdf6640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:27.709 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.707+0000 7fc38cdf6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:27.709 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.707+0000 7fc38cdf6640 10 monclient(hunting): picked mon.noname-a con 0x7fc3880b4d60 addr v1:172.21.15.156:6789/0 2024-03-25T18:18:27.709 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.707+0000 7fc38cdf6640 10 monclient(hunting): picked mon.noname-b con 0x7fc3880b1730 addr v1:172.21.15.156:6790/0 2024-03-25T18:18:27.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.707+0000 7fc38cdf6640 10 monclient(hunting): picked mon.noname-c con 0x7fc38814be90 addr v1:172.21.15.194:6789/0 2024-03-25T18:18:27.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.707+0000 7fc38cdf6640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3880af940 con 0x7fc3880b4d60 2024-03-25T18:18:27.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.707+0000 7fc38cdf6640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3880b0a70 con 0x7fc38814be90 2024-03-25T18:18:27.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.707+0000 7fc38cdf6640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3880b0d50 con 0x7fc3880b1730 2024-03-25T18:18:27.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.707+0000 7fc38cdf6640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:27.710 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.707+0000 7fc38cdf6640 10 monclient(hunting): authenticate will time out at 7589.019094s 2024-03-25T18:18:27.717 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.715+0000 7fc386d9d640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fc38814be90 0x7fc38814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:37828/0 (socket says 172.21.15.156:37828) 2024-03-25T18:18:27.717 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.715+0000 7fc386d9d640 1 -- 172.21.15.156:0/3441629319 learned_addr learned my addr 172.21.15.156:0/3441629319 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:18:27.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.729+0000 7fc38559a640 1 -- 172.21.15.156:0/3441629319 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1427706996 0 0) 0x7fc3880b0a70 con 0x7fc38814be90 2024-03-25T18:18:27.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.729+0000 7fc38559a640 10 monclient(hunting): _negotiate 2024-03-25T18:18:27.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.729+0000 7fc38559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:27.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.729+0000 7fc38559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:27.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.729+0000 7fc38559a640 10 monclient(hunting): my global_id is 4684 2024-03-25T18:18:27.732 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.729+0000 7fc38559a640 1 -- 172.21.15.156:0/3441629319 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc370003520 con 0x7fc38814be90 2024-03-25T18:18:27.736 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.734+0000 7fc38559a640 1 -- 172.21.15.156:0/3441629319 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337445857 0 0) 0x7fc3880b0d50 con 0x7fc3880b1730 2024-03-25T18:18:27.736 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.734+0000 7fc38559a640 10 monclient(hunting): _negotiate 2024-03-25T18:18:27.736 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.734+0000 7fc38559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:27.736 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.734+0000 7fc38559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:27.736 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.734+0000 7fc38559a640 10 monclient(hunting): my global_id is 4688 2024-03-25T18:18:27.736 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.734+0000 7fc38559a640 1 -- 172.21.15.156:0/3441629319 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3880b0a70 con 0x7fc3880b1730 2024-03-25T18:18:27.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.740+0000 7fc38559a640 1 -- 172.21.15.156:0/3441629319 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1111822281 0 0) 0x7fc370003520 con 0x7fc38814be90 2024-03-25T18:18:27.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.740+0000 7fc38559a640 1 -- 172.21.15.156:0/3441629319 >> v1:172.21.15.156:6790/0 conn(0x7fc3880b1730 legacy=0x7fc3880b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:27.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.740+0000 7fc38559a640 1 -- 172.21.15.156:0/3441629319 >> v1:172.21.15.156:6789/0 conn(0x7fc3880b4d60 legacy=0x7fc38813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:27.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.740+0000 7fc38559a640 10 monclient: _finish_hunting 0 2024-03-25T18:18:27.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.740+0000 7fc38559a640 1 monclient: found mon.noname-c 2024-03-25T18:18:27.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.741+0000 7fc38559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:27.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.741+0000 7fc38559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:18:27.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.741+0000 7fc38559a640 1 -- 172.21.15.156:0/3441629319 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3880b1030 con 0x7fc38814be90 2024-03-25T18:18:27.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.741+0000 7fc38559a640 10 monclient: _finish_auth 0 2024-03-25T18:18:27.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.741+0000 7fc38559a640 10 monclient: _check_auth_tickets 2024-03-25T18:18:27.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.741+0000 7fc38559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:27.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.741+0000 7fc38559a640 1 -- 172.21.15.156:0/3441629319 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc380001a10 con 0x7fc38814be90 2024-03-25T18:18:27.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.741+0000 7fc38cdf6640 5 monclient: authenticate success, global_id 4684 2024-03-25T18:18:27.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.741+0000 7fc38cdf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:27.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.741+0000 7fc38559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:27.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.741+0000 7fc38559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:27.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.741+0000 7fc38559a640 10 monclient: dump: 2024-03-25T18:18:27.744 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:27.745 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:27.745 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:27.745 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:27.745 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:27.745 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:27.745 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:27.745 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:27.745 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:27.745 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:27.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.741+0000 7fc38cdf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:27.759 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.757+0000 7fc38559a640 1 -- 172.21.15.156:0/3441629319 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc380001b70 con 0x7fc38814be90 2024-03-25T18:18:27.759 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.757+0000 7fc38559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:27.759 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.757+0000 7fc38559a640 1 -- 172.21.15.156:0/3441629319 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc380001f80 con 0x7fc38814be90 2024-03-25T18:18:27.759 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.757+0000 7fc38cdf6640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:27.759 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.757+0000 7fc38cdf6640 4 set_mon_vals no callback set 2024-03-25T18:18:27.760 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.757+0000 7fc38559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:27.760 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.757+0000 7fc38559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:27.760 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.757+0000 7fc38559a640 10 monclient: dump: 2024-03-25T18:18:27.760 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:27.760 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:27.760 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:27.760 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:27.760 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:27.760 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:27.760 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:27.760 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:27.760 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:27.760 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:27.761 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.758+0000 7fc38cdf6640 10 monclient: shutdown 2024-03-25T18:18:27.761 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.758+0000 7fc38cdf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:27.761 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.758+0000 7fc38cdf6640 1 -- 172.21.15.156:0/3441629319 >> v1:172.21.15.194:6789/0 conn(0x7fc38814be90 legacy=0x7fc38814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:27.761 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.758+0000 7fc38cdf6640 1 -- 172.21.15.156:0/3441629319 shutdown_connections 2024-03-25T18:18:27.762 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.758+0000 7fc38cdf6640 1 -- 172.21.15.156:0/3441629319 >> 172.21.15.156:0/3441629319 conn(0x7fc3880617a0 msgr2=0x7fc388061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:27.762 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.758+0000 7fc38cdf6640 1 -- 172.21.15.156:0/3441629319 shutdown_connections 2024-03-25T18:18:27.762 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.758+0000 7fc38cdf6640 1 -- 172.21.15.156:0/3441629319 wait complete. 2024-03-25T18:18:27.762 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.759+0000 7fc38cdf6640 10 monclient: build_initial_monmap 2024-03-25T18:18:27.762 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.759+0000 7fc38cdf6640 10 monclient: monmap: 2024-03-25T18:18:27.762 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:18:27.762 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:27.762 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:18:27.762 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:18:27.762 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:27.762 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:18:27.762 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:18:27.762 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:18:27.763 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:18:27.763 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:27.764 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.760+0000 7fc38cdf6640 1 librados: starting msgr at 2024-03-25T18:18:27.764 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.760+0000 7fc38cdf6640 1 librados: starting objecter 2024-03-25T18:18:27.764 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.760+0000 7fc38cdf6640 1 Processor -- start 2024-03-25T18:18:27.764 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.760+0000 7fc38cdf6640 1 -- start start 2024-03-25T18:18:27.764 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.760+0000 7fc38cdf6640 1 librados: setting wanted keys 2024-03-25T18:18:27.764 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.760+0000 7fc38cdf6640 1 librados: calling monclient init 2024-03-25T18:18:27.764 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.760+0000 7fc38cdf6640 10 monclient: init 2024-03-25T18:18:27.765 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.761+0000 7fc38cdf6640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:27.765 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.761+0000 7fc38cdf6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:27.765 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.761+0000 7fc38cdf6640 10 monclient(hunting): picked mon.noname-a con 0x7fc3880b1730 addr v1:172.21.15.156:6789/0 2024-03-25T18:18:27.765 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.761+0000 7fc38cdf6640 10 monclient(hunting): picked mon.noname-b con 0x7fc3880b4d60 addr v1:172.21.15.194:6789/0 2024-03-25T18:18:27.765 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.761+0000 7fc38cdf6640 10 monclient(hunting): picked mon.noname-c con 0x7fc38814be90 addr v1:172.21.15.156:6790/0 2024-03-25T18:18:27.765 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.761+0000 7fc38cdf6640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3881385a0 con 0x7fc3880b1730 2024-03-25T18:18:27.765 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.761+0000 7fc38cdf6640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc388139840 con 0x7fc3880b4d60 2024-03-25T18:18:27.765 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.761+0000 7fc38cdf6640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc388138080 con 0x7fc38814be90 2024-03-25T18:18:27.765 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.761+0000 7fc38cdf6640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:27.765 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.761+0000 7fc38cdf6640 10 monclient(hunting): authenticate will time out at 7589.073320s 2024-03-25T18:18:27.773 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.771+0000 7fc38659c640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fc3880b1730 0x7fc388131e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:51166/0 (socket says 172.21.15.156:51166) 2024-03-25T18:18:27.773 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.771+0000 7fc38659c640 1 -- 172.21.15.156:0/889112384 learned_addr learned my addr 172.21.15.156:0/889112384 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:18:27.776 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.774+0000 7fc376ffd640 10 client.?.objecter ms_handle_connect 0x7fc3880b1730 2024-03-25T18:18:27.776 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.774+0000 7fc376ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:27.784 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.781+0000 7fc376ffd640 10 client.?.objecter ms_handle_connect 0x7fc38814be90 2024-03-25T18:18:27.784 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.781+0000 7fc376ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:27.784 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.782+0000 7fc376ffd640 10 client.?.objecter ms_handle_connect 0x7fc3880b4d60 2024-03-25T18:18:27.784 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.782+0000 7fc376ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:27.791 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.789+0000 7fc376ffd640 1 -- 172.21.15.156:0/889112384 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731067123 0 0) 0x7fc388138080 con 0x7fc38814be90 2024-03-25T18:18:27.791 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.789+0000 7fc376ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:27.791 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.789+0000 7fc376ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:27.792 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.789+0000 7fc376ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:27.792 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.789+0000 7fc376ffd640 10 monclient(hunting): my global_id is 4691 2024-03-25T18:18:27.792 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.789+0000 7fc376ffd640 1 -- 172.21.15.156:0/889112384 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc368004eb0 con 0x7fc38814be90 2024-03-25T18:18:27.793 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.790+0000 7fc376ffd640 1 -- 172.21.15.156:0/889112384 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2202806179 0 0) 0x7fc3881385a0 con 0x7fc3880b1730 2024-03-25T18:18:27.793 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.791+0000 7fc376ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:27.793 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.791+0000 7fc376ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:27.793 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.791+0000 7fc376ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:27.793 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.791+0000 7fc376ffd640 10 monclient(hunting): my global_id is 4683 2024-03-25T18:18:27.793 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.791+0000 7fc376ffd640 1 -- 172.21.15.156:0/889112384 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc388138080 con 0x7fc3880b1730 2024-03-25T18:18:27.798 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.796+0000 7fc376ffd640 1 -- 172.21.15.156:0/889112384 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154120148 0 0) 0x7fc388139840 con 0x7fc3880b4d60 2024-03-25T18:18:27.799 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.796+0000 7fc376ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:27.799 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.796+0000 7fc376ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:27.799 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.796+0000 7fc376ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:27.799 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.796+0000 7fc376ffd640 10 monclient(hunting): my global_id is 4687 2024-03-25T18:18:27.799 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.796+0000 7fc376ffd640 1 -- 172.21.15.156:0/889112384 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3881385a0 con 0x7fc3880b4d60 2024-03-25T18:18:27.805 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.803+0000 7fc376ffd640 1 -- 172.21.15.156:0/889112384 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3184918786 0 0) 0x7fc388138080 con 0x7fc3880b1730 2024-03-25T18:18:27.805 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.803+0000 7fc376ffd640 1 -- 172.21.15.156:0/889112384 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc388139840 con 0x7fc3880b1730 2024-03-25T18:18:27.805 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.803+0000 7fc376ffd640 1 -- 172.21.15.156:0/889112384 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc378001970 con 0x7fc3880b1730 2024-03-25T18:18:27.806 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.803+0000 7fc376ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:27.806 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.803+0000 7fc376ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:27.806 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.803+0000 7fc376ffd640 10 monclient(hunting): dump: 2024-03-25T18:18:27.806 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:27.806 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:27.806 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:27.806 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:27.806 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:27.806 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:27.806 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:27.806 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:27.806 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:27.806 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:27.807 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.803+0000 7fc376ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:27.807 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.805+0000 7fc376ffd640 1 -- 172.21.15.156:0/889112384 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 119216381 0 0) 0x7fc368004eb0 con 0x7fc38814be90 2024-03-25T18:18:27.807 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.805+0000 7fc376ffd640 1 -- 172.21.15.156:0/889112384 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc388138080 con 0x7fc38814be90 2024-03-25T18:18:27.807 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.805+0000 7fc376ffd640 1 -- 172.21.15.156:0/889112384 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc380001ec0 con 0x7fc38814be90 2024-03-25T18:18:27.807 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.805+0000 7fc376ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:27.807 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.805+0000 7fc376ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:27.807 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.805+0000 7fc376ffd640 10 monclient(hunting): dump: 2024-03-25T18:18:27.807 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:27.808 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:27.808 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:27.808 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:27.808 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:27.808 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:27.808 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:27.808 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:27.808 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:27.808 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:27.810 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.807+0000 7fc376ffd640 1 -- 172.21.15.156:0/889112384 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2540503327 0 0) 0x7fc388139840 con 0x7fc3880b1730 2024-03-25T18:18:27.810 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.807+0000 7fc376ffd640 1 -- 172.21.15.156:0/889112384 >> v1:172.21.15.156:6790/0 conn(0x7fc38814be90 legacy=0x7fc38813b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:27.810 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.807+0000 7fc376ffd640 1 -- 172.21.15.156:0/889112384 >> v1:172.21.15.194:6789/0 conn(0x7fc3880b4d60 legacy=0x7fc38813b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:27.810 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.807+0000 7fc376ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:27.810 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.807+0000 7fc376ffd640 1 monclient: found mon.a 2024-03-25T18:18:27.810 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.807+0000 7fc376ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:27.811 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.807+0000 7fc376ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:18:27.811 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.807+0000 7fc376ffd640 1 -- 172.21.15.156:0/889112384 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc388138210 con 0x7fc3880b1730 2024-03-25T18:18:27.811 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.808+0000 7fc376ffd640 10 monclient: _finish_auth 0 2024-03-25T18:18:27.811 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.808+0000 7fc376ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:27.811 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.808+0000 7fc376ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:27.811 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.808+0000 7fc38cdf6640 5 monclient: authenticate success, global_id 4683 2024-03-25T18:18:27.811 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.808+0000 7fc38cdf6640 10 monclient: _renew_subs 2024-03-25T18:18:27.811 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.808+0000 7fc38cdf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:18:27.811 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.808+0000 7fc38cdf6640 1 -- 172.21.15.156:0/889112384 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc388138dd0 con 0x7fc3880b1730 2024-03-25T18:18:27.811 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.808+0000 7fc38cdf6640 10 client.4683.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:27.811 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.808+0000 7fc38cdf6640 10 monclient: _renew_subs 2024-03-25T18:18:27.811 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.808+0000 7fc38cdf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:18:27.811 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.808+0000 7fc38cdf6640 1 -- 172.21.15.156:0/889112384 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc388132700 con 0x7fc3880b1730 2024-03-25T18:18:27.811 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.808+0000 7fc38cdf6640 1 librados: init done 2024-03-25T18:18:27.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.809+0000 7fc38cdf6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:27.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.809+0000 7fc38cdf6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:27.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.809+0000 7fc38cdf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:18:27.812 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.809+0000 7fc38cdf6640 1 -- 172.21.15.156:0/889112384 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc388138400 con 0x7fc3880b1730 2024-03-25T18:18:27.813 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.811+0000 7fc376ffd640 1 -- 172.21.15.156:0/889112384 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc378003610 con 0x7fc3880b1730 2024-03-25T18:18:27.813 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.811+0000 7fc376ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:27.813 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:27.811+0000 7fc384d99640 4 set_mon_vals no callback set 2024-03-25T18:18:28.117 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.115+0000 7fc376ffd640 1 -- 172.21.15.156:0/889112384 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fc378003800 con 0x7fc3880b1730 2024-03-25T18:18:28.117 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.115+0000 7fc376ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:28.118 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.115+0000 7fc376ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:28.118 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.115+0000 7fc376ffd640 10 monclient: dump: 2024-03-25T18:18:28.118 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:28.118 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:28.118 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:28.118 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:28.118 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:28.118 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:28.118 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:28.118 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:28.118 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:28.118 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:28.119 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.116+0000 7fc376ffd640 1 -- 172.21.15.156:0/889112384 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7fc378003960 con 0x7fc3880b1730 2024-03-25T18:18:28.119 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.117+0000 7fc385d9b640 10 monclient: get_auth_request con 0x7fc36806df20 auth_method 0 2024-03-25T18:18:28.119 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.117+0000 7fc376ffd640 10 client.4683.objecter ms_handle_connect 0x7fc36806df20 2024-03-25T18:18:28.125 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.122+0000 7fc376ffd640 1 -- 172.21.15.156:0/889112384 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3381691338 0 0) 0x7fc378054680 con 0x7fc3880b1730 2024-03-25T18:18:28.125 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.122+0000 7fc376ffd640 10 client.4683.objecter ms_dispatch 0x7fc388000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:18:28.125 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.122+0000 7fc376ffd640 3 client.4683.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:18:28.125 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.122+0000 7fc376ffd640 3 client.4683.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:18:28.125 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.122+0000 7fc376ffd640 20 client.4683.objecter dump_active .. 0 homeless 2024-03-25T18:18:28.125 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.122+0000 7fc376ffd640 1 -- 172.21.15.156:0/889112384 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc37804ffc0 con 0x7fc3880b1730 2024-03-25T18:18:28.125 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.122+0000 7fc376ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:28.125 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.122+0000 7fc376ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:28.354 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.352+0000 7fc38cdf6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:28.354 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.352+0000 7fc38cdf6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:28.354 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.352+0000 7fc38cdf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:18:28.355 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.352+0000 7fc38cdf6640 1 -- 172.21.15.156:0/889112384 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc388138730 con 0x7fc3880b1730 2024-03-25T18:18:28.374 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.372+0000 7fc376ffd640 1 -- 172.21.15.156:0/889112384 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1441834839) 0x7fc3780520c0 con 0x7fc3880b1730 2024-03-25T18:18:28.374 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.372+0000 7fc376ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:28.374 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.372+0000 7fc376ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:28.374 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:18:28.375 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":11,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:18:20.029906+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:18:20.029906+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:28.377 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.374+0000 7fc38cdf6640 10 librados: watch_flush enter 2024-03-25T18:18:28.377 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.374+0000 7fc38cdf6640 10 librados: watch_flush exit 2024-03-25T18:18:28.377 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.374+0000 7fc38cdf6640 20 client.4683.objecter shutdown clearing up homeless session... 2024-03-25T18:18:28.377 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.374+0000 7fc38cdf6640 10 client.4683.objecter successfully canceled tick 2024-03-25T18:18:28.377 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.374+0000 7fc38cdf6640 1 -- 172.21.15.156:0/889112384 >> v1:172.21.15.194:6800/2946427713 conn(0x7fc36806df20 legacy=0x7fc3680703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:28.377 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.374+0000 7fc38cdf6640 10 monclient: shutdown 2024-03-25T18:18:28.377 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.374+0000 7fc38cdf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:28.377 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.374+0000 7fc38cdf6640 1 -- 172.21.15.156:0/889112384 >> v1:172.21.15.156:6789/0 conn(0x7fc3880b1730 legacy=0x7fc388131e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:28.377 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.374+0000 7fc38cdf6640 1 -- 172.21.15.156:0/889112384 shutdown_connections 2024-03-25T18:18:28.377 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.374+0000 7fc38cdf6640 1 -- 172.21.15.156:0/889112384 >> 172.21.15.156:0/889112384 conn(0x7fc3880617a0 msgr2=0x7fc38813c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:28.378 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.374+0000 7fc38cdf6640 1 -- 172.21.15.156:0/889112384 shutdown_connections 2024-03-25T18:18:28.378 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.374+0000 7fc38cdf6640 1 -- 172.21.15.156:0/889112384 wait complete. 2024-03-25T18:18:28.378 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:28.374+0000 7fc38cdf6640 1 librados: shutdown 2024-03-25T18:18:28.387 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:35.195 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:35.307 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.306+0000 7f4b7e0fb640 10 monclient: get_monmap_and_config 2024-03-25T18:18:35.307 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.306+0000 7f4b7e0fb640 10 monclient: build_initial_monmap 2024-03-25T18:18:35.307 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.306+0000 7f4b7e0fb640 10 monclient: monmap: 2024-03-25T18:18:35.308 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:18:35.308 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:35.308 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:18:35.307498+0000 2024-03-25T18:18:35.308 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:18:35.307498+0000 2024-03-25T18:18:35.308 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:35.308 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:35.308 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:18:35.308 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:18:35.308 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:18:35.308 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:35.310 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.307+0000 7f4b7e0fb640 1 Processor -- start 2024-03-25T18:18:35.310 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.307+0000 7f4b7e0fb640 1 -- start start 2024-03-25T18:18:35.310 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.307+0000 7f4b7e0fb640 10 monclient: init 2024-03-25T18:18:35.310 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.308+0000 7f4b7e0fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:35.310 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.308+0000 7f4b7e0fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:18:35.310 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.308+0000 7f4b7e0fb640 10 monclient(hunting): picked mon.noname-c con 0x7f4b780a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:18:35.310 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.308+0000 7f4b7e0fb640 10 monclient(hunting): picked mon.noname-b con 0x7f4b7813f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:18:35.310 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.308+0000 7f4b7e0fb640 10 monclient(hunting): picked mon.noname-a con 0x7f4b781430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:18:35.310 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.308+0000 7f4b7e0fb640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b78142520 con 0x7f4b781430c0 2024-03-25T18:18:35.310 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.308+0000 7f4b7e0fb640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b780b28a0 con 0x7f4b780a98f0 2024-03-25T18:18:35.310 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.308+0000 7f4b7e0fb640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b780b2a30 con 0x7f4b7813f710 2024-03-25T18:18:35.310 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.308+0000 7f4b7e0fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:35.310 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.308+0000 7f4b7e0fb640 10 monclient(hunting): authenticate will time out at 7596.619354s 2024-03-25T18:18:35.311 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.310+0000 7f4b777fe640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f4b780a98f0 0x7f4b780a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:50910/0 (socket says 172.21.15.156:50910) 2024-03-25T18:18:35.311 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.310+0000 7f4b777fe640 1 -- 172.21.15.156:0/1152150507 learned_addr learned my addr 172.21.15.156:0/1152150507 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:18:35.319 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.318+0000 7f4b767fc640 1 -- 172.21.15.156:0/1152150507 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 530433965 0 0) 0x7f4b780b28a0 con 0x7f4b780a98f0 2024-03-25T18:18:35.319 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.318+0000 7f4b767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:35.319 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.318+0000 7f4b767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:35.320 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.318+0000 7f4b767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:35.320 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.318+0000 7f4b767fc640 10 monclient(hunting): my global_id is 4690 2024-03-25T18:18:35.320 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.318+0000 7f4b767fc640 1 -- 172.21.15.156:0/1152150507 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b60003520 con 0x7f4b780a98f0 2024-03-25T18:18:35.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.386+0000 7f4b767fc640 1 -- 172.21.15.156:0/1152150507 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1846037150 0 0) 0x7f4b78142520 con 0x7f4b781430c0 2024-03-25T18:18:35.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.386+0000 7f4b767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:35.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.386+0000 7f4b767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:35.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.386+0000 7f4b767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:35.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.386+0000 7f4b767fc640 10 monclient(hunting): my global_id is 4686 2024-03-25T18:18:35.387 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.386+0000 7f4b767fc640 1 -- 172.21.15.156:0/1152150507 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b780b28a0 con 0x7f4b781430c0 2024-03-25T18:18:35.397 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.396+0000 7f4b767fc640 1 -- 172.21.15.156:0/1152150507 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1160600100 0 0) 0x7f4b780b28a0 con 0x7f4b781430c0 2024-03-25T18:18:35.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.396+0000 7f4b767fc640 1 -- 172.21.15.156:0/1152150507 >> v1:172.21.15.156:6790/0 conn(0x7f4b7813f710 legacy=0x7f4b781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:35.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.396+0000 7f4b767fc640 1 -- 172.21.15.156:0/1152150507 >> v1:172.21.15.194:6789/0 conn(0x7f4b780a98f0 legacy=0x7f4b780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:35.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.396+0000 7f4b767fc640 10 monclient: _finish_hunting 0 2024-03-25T18:18:35.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.396+0000 7f4b767fc640 1 monclient: found mon.noname-a 2024-03-25T18:18:35.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.396+0000 7f4b767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:35.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.396+0000 7f4b767fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:18:35.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.396+0000 7f4b767fc640 1 -- 172.21.15.156:0/1152150507 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b780b44c0 con 0x7f4b781430c0 2024-03-25T18:18:35.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.396+0000 7f4b767fc640 10 monclient: _finish_auth 0 2024-03-25T18:18:35.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.396+0000 7f4b767fc640 10 monclient: _check_auth_tickets 2024-03-25T18:18:35.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.396+0000 7f4b767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:35.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.396+0000 7f4b767fc640 1 -- 172.21.15.156:0/1152150507 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f4b700019f0 con 0x7f4b781430c0 2024-03-25T18:18:35.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.396+0000 7f4b767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:35.399 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.396+0000 7f4b767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:35.399 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.396+0000 7f4b767fc640 10 monclient: dump: 2024-03-25T18:18:35.399 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:35.399 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:35.399 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:35.399 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:35.399 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:35.399 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:35.399 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:35.399 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:35.399 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:35.399 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:35.399 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.396+0000 7f4b7e0fb640 5 monclient: authenticate success, global_id 4686 2024-03-25T18:18:35.399 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.396+0000 7f4b7e0fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:35.425 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.424+0000 7f4b767fc640 1 -- 172.21.15.156:0/1152150507 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b70001b50 con 0x7f4b781430c0 2024-03-25T18:18:35.425 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.424+0000 7f4b767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:35.425 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.424+0000 7f4b767fc640 1 -- 172.21.15.156:0/1152150507 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f4b70001f20 con 0x7f4b781430c0 2024-03-25T18:18:35.425 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.424+0000 7f4b7e0fb640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:35.425 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.424+0000 7f4b7e0fb640 4 set_mon_vals no callback set 2024-03-25T18:18:35.425 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.424+0000 7f4b767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:35.425 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.424+0000 7f4b767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:35.426 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.424+0000 7f4b767fc640 10 monclient: dump: 2024-03-25T18:18:35.426 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:35.426 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:35.426 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:35.426 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:35.426 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:35.426 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:35.426 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:35.426 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:35.426 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:35.426 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:35.428 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.425+0000 7f4b7e0fb640 10 monclient: shutdown 2024-03-25T18:18:35.428 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.425+0000 7f4b7e0fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:35.429 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.425+0000 7f4b7e0fb640 1 -- 172.21.15.156:0/1152150507 >> v1:172.21.15.156:6789/0 conn(0x7f4b781430c0 legacy=0x7f4b7814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:35.429 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.425+0000 7f4b7e0fb640 1 -- 172.21.15.156:0/1152150507 shutdown_connections 2024-03-25T18:18:35.429 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.425+0000 7f4b7e0fb640 1 -- 172.21.15.156:0/1152150507 >> 172.21.15.156:0/1152150507 conn(0x7f4b780b7ea0 msgr2=0x7f4b780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:35.429 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.425+0000 7f4b7e0fb640 1 -- 172.21.15.156:0/1152150507 shutdown_connections 2024-03-25T18:18:35.429 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.425+0000 7f4b7e0fb640 1 -- 172.21.15.156:0/1152150507 wait complete. 2024-03-25T18:18:35.429 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.426+0000 7f4b7e0fb640 10 monclient: build_initial_monmap 2024-03-25T18:18:35.429 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.426+0000 7f4b7e0fb640 10 monclient: monmap: 2024-03-25T18:18:35.429 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:18:35.429 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:35.429 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:18:35.429 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:18:35.429 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:35.429 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:18:35.429 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:18:35.429 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:18:35.430 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:18:35.430 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:35.430 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.426+0000 7f4b7e0fb640 1 librados: starting msgr at 2024-03-25T18:18:35.430 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.426+0000 7f4b7e0fb640 1 librados: starting objecter 2024-03-25T18:18:35.430 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.426+0000 7f4b7e0fb640 1 Processor -- start 2024-03-25T18:18:35.430 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.426+0000 7f4b7e0fb640 1 -- start start 2024-03-25T18:18:35.430 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.427+0000 7f4b7e0fb640 1 librados: setting wanted keys 2024-03-25T18:18:35.430 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.427+0000 7f4b7e0fb640 1 librados: calling monclient init 2024-03-25T18:18:35.430 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.427+0000 7f4b7e0fb640 10 monclient: init 2024-03-25T18:18:35.430 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.427+0000 7f4b7e0fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:35.430 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.427+0000 7f4b7e0fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:35.430 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.427+0000 7f4b7e0fb640 10 monclient(hunting): picked mon.noname-a con 0x7f4b780a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:18:35.430 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.427+0000 7f4b7e0fb640 10 monclient(hunting): picked mon.noname-c con 0x7f4b7813f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:18:35.430 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.427+0000 7f4b7e0fb640 10 monclient(hunting): picked mon.noname-b con 0x7f4b781430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:18:35.431 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.427+0000 7f4b7e0fb640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b78138a60 con 0x7f4b780a98f0 2024-03-25T18:18:35.431 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.427+0000 7f4b7e0fb640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b781397c0 con 0x7f4b781430c0 2024-03-25T18:18:35.431 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.427+0000 7f4b7e0fb640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b781d7810 con 0x7f4b7813f710 2024-03-25T18:18:35.431 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.427+0000 7f4b7e0fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:35.431 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.427+0000 7f4b7e0fb640 10 monclient(hunting): authenticate will time out at 7596.738384s 2024-03-25T18:18:35.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.442+0000 7f4b76ffd640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f4b7813f710 0x7f4b781d7d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:52496/0 (socket says 172.21.15.156:52496) 2024-03-25T18:18:35.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.442+0000 7f4b76ffd640 1 -- 172.21.15.156:0/3275500876 learned_addr learned my addr 172.21.15.156:0/3275500876 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:18:35.446 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.445+0000 7f4b5ffff640 10 client.?.objecter ms_handle_connect 0x7f4b7813f710 2024-03-25T18:18:35.446 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.445+0000 7f4b5ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:35.448 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.447+0000 7f4b5ffff640 10 client.?.objecter ms_handle_connect 0x7f4b780a98f0 2024-03-25T18:18:35.448 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.447+0000 7f4b5ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:35.453 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.452+0000 7f4b5ffff640 10 client.?.objecter ms_handle_connect 0x7f4b781430c0 2024-03-25T18:18:35.453 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.452+0000 7f4b5ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:35.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.456+0000 7f4b5ffff640 1 -- 172.21.15.156:0/3275500876 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1370789428 0 0) 0x7f4b781d7810 con 0x7f4b7813f710 2024-03-25T18:18:35.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.457+0000 7f4b5ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:35.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.457+0000 7f4b5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:35.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.457+0000 7f4b5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:35.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.457+0000 7f4b5ffff640 10 monclient(hunting): my global_id is 4697 2024-03-25T18:18:35.458 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.457+0000 7f4b5ffff640 1 -- 172.21.15.156:0/3275500876 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b64004eb0 con 0x7f4b7813f710 2024-03-25T18:18:35.461 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.460+0000 7f4b5ffff640 1 -- 172.21.15.156:0/3275500876 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3714394149 0 0) 0x7f4b78138a60 con 0x7f4b780a98f0 2024-03-25T18:18:35.461 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.460+0000 7f4b5ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:35.461 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.460+0000 7f4b5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:35.461 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.460+0000 7f4b5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:35.462 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.460+0000 7f4b5ffff640 10 monclient(hunting): my global_id is 4689 2024-03-25T18:18:35.462 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.460+0000 7f4b5ffff640 1 -- 172.21.15.156:0/3275500876 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b781d7810 con 0x7f4b780a98f0 2024-03-25T18:18:35.467 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.466+0000 7f4b5ffff640 1 -- 172.21.15.156:0/3275500876 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1408633930 0 0) 0x7f4b781397c0 con 0x7f4b781430c0 2024-03-25T18:18:35.467 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.466+0000 7f4b5ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:35.468 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.466+0000 7f4b5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:35.468 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.466+0000 7f4b5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:35.468 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.466+0000 7f4b5ffff640 10 monclient(hunting): my global_id is 4693 2024-03-25T18:18:35.468 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.466+0000 7f4b5ffff640 1 -- 172.21.15.156:0/3275500876 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b78138a60 con 0x7f4b781430c0 2024-03-25T18:18:35.475 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.474+0000 7f4b5ffff640 1 -- 172.21.15.156:0/3275500876 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1353447262 0 0) 0x7f4b64004eb0 con 0x7f4b7813f710 2024-03-25T18:18:35.475 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.474+0000 7f4b5ffff640 1 -- 172.21.15.156:0/3275500876 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b781397c0 con 0x7f4b7813f710 2024-03-25T18:18:35.475 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.474+0000 7f4b5ffff640 1 -- 172.21.15.156:0/3275500876 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f4b6c0019d0 con 0x7f4b7813f710 2024-03-25T18:18:35.475 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.474+0000 7f4b5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:35.475 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.474+0000 7f4b5ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:35.476 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.474+0000 7f4b5ffff640 10 monclient(hunting): dump: 2024-03-25T18:18:35.476 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:35.476 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:35.476 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:35.476 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:35.476 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:35.476 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:35.476 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:35.476 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:35.476 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:35.476 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:35.476 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.474+0000 7f4b5ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:35.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.485+0000 7f4b5ffff640 1 -- 172.21.15.156:0/3275500876 <== mon.2 v1:172.21.15.156:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1611600740 0 0) 0x7f4b781397c0 con 0x7f4b7813f710 2024-03-25T18:18:35.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.485+0000 7f4b5ffff640 1 -- 172.21.15.156:0/3275500876 >> v1:172.21.15.194:6789/0 conn(0x7f4b781430c0 legacy=0x7f4b781e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:35.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.485+0000 7f4b5ffff640 1 -- 172.21.15.156:0/3275500876 >> v1:172.21.15.156:6789/0 conn(0x7f4b780a98f0 legacy=0x7f4b78138350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:35.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.485+0000 7f4b5ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:18:35.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.485+0000 7f4b5ffff640 1 monclient: found mon.c 2024-03-25T18:18:35.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.485+0000 7f4b5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:35.487 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.485+0000 7f4b5ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:18:35.488 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.485+0000 7f4b5ffff640 1 -- 172.21.15.156:0/3275500876 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b781d51f0 con 0x7f4b7813f710 2024-03-25T18:18:35.488 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.485+0000 7f4b5ffff640 10 monclient: _finish_auth 0 2024-03-25T18:18:35.488 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.485+0000 7f4b5ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:18:35.488 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.485+0000 7f4b5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:35.488 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.485+0000 7f4b7e0fb640 5 monclient: authenticate success, global_id 4697 2024-03-25T18:18:35.488 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.485+0000 7f4b7e0fb640 10 monclient: _renew_subs 2024-03-25T18:18:35.488 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.486+0000 7f4b7e0fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:18:35.488 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.486+0000 7f4b7e0fb640 1 -- 172.21.15.156:0/3275500876 --> v1:172.21.15.156:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b781d8440 con 0x7f4b7813f710 2024-03-25T18:18:35.488 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.486+0000 7f4b7e0fb640 10 client.4697.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:35.488 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.486+0000 7f4b7e0fb640 10 monclient: _renew_subs 2024-03-25T18:18:35.488 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.486+0000 7f4b7e0fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:18:35.488 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.486+0000 7f4b7e0fb640 1 -- 172.21.15.156:0/3275500876 --> v1:172.21.15.156:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b78138eb0 con 0x7f4b7813f710 2024-03-25T18:18:35.489 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.486+0000 7f4b7e0fb640 1 librados: init done 2024-03-25T18:18:35.489 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.487+0000 7f4b7e0fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:35.489 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.487+0000 7f4b7e0fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:35.489 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.487+0000 7f4b7e0fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:18:35.489 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.487+0000 7f4b7e0fb640 1 -- 172.21.15.156:0/3275500876 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b781d79a0 con 0x7f4b7813f710 2024-03-25T18:18:35.782 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.781+0000 7f4b5ffff640 1 -- 172.21.15.156:0/3275500876 <== mon.2 v1:172.21.15.156:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b6c0035f0 con 0x7f4b7813f710 2024-03-25T18:18:35.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.781+0000 7f4b5ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:35.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.781+0000 7f4b5ffff640 1 -- 172.21.15.156:0/3275500876 <== mon.2 v1:172.21.15.156:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f4b6c003a00 con 0x7f4b7813f710 2024-03-25T18:18:35.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.781+0000 7f4b5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:35.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.781+0000 7f4b5ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:35.783 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.781+0000 7f4b5ffff640 10 monclient: dump: 2024-03-25T18:18:35.783 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:35.783 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:35.783 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:35.783 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:35.783 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:35.783 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:35.783 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:35.783 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:35.783 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:35.783 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:35.784 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.781+0000 7f4b75ffb640 4 set_mon_vals no callback set 2024-03-25T18:18:35.784 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.782+0000 7f4b5ffff640 1 -- 172.21.15.156:0/3275500876 <== mon.2 v1:172.21.15.156:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f4b6c003c20 con 0x7f4b7813f710 2024-03-25T18:18:35.784 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.783+0000 7f4b777fe640 10 monclient: get_auth_request con 0x7f4b6406ddf0 auth_method 0 2024-03-25T18:18:35.784 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.783+0000 7f4b5ffff640 10 client.4697.objecter ms_handle_connect 0x7f4b6406ddf0 2024-03-25T18:18:35.791 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.789+0000 7f4b5ffff640 1 -- 172.21.15.156:0/3275500876 <== mon.2 v1:172.21.15.156:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3381691338 0 0) 0x7f4b6c054820 con 0x7f4b7813f710 2024-03-25T18:18:35.791 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.789+0000 7f4b5ffff640 10 client.4697.objecter ms_dispatch 0x7f4b78000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:18:35.791 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.789+0000 7f4b5ffff640 3 client.4697.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:18:35.791 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.789+0000 7f4b5ffff640 3 client.4697.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:18:35.791 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.790+0000 7f4b5ffff640 20 client.4697.objecter dump_active .. 0 homeless 2024-03-25T18:18:35.791 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.790+0000 7f4b5ffff640 1 -- 172.21.15.156:0/3275500876 <== mon.2 v1:172.21.15.156:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b6c050330 con 0x7f4b7813f710 2024-03-25T18:18:35.791 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.790+0000 7f4b5ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:35.791 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:35.790+0000 7f4b5ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:36.038 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:36.037+0000 7f4b7e0fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:36.038 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:36.037+0000 7f4b7e0fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:36.038 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:36.037+0000 7f4b7e0fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:18:36.038 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:36.037+0000 7f4b7e0fb640 1 -- 172.21.15.156:0/3275500876 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b780627b0 con 0x7f4b7813f710 2024-03-25T18:18:36.099 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:36.097+0000 7f4b5ffff640 1 -- 172.21.15.156:0/3275500876 <== mon.2 v1:172.21.15.156:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1441834839) 0x7f4b6c052260 con 0x7f4b7813f710 2024-03-25T18:18:36.099 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:36.097+0000 7f4b5ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:36.099 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:36.097+0000 7f4b5ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:36.099 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:18:36.099 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":11,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:18:20.029906+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:18:20.029906+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:36.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:36.099+0000 7f4b7e0fb640 10 librados: watch_flush enter 2024-03-25T18:18:36.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:36.099+0000 7f4b7e0fb640 10 librados: watch_flush exit 2024-03-25T18:18:36.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:36.099+0000 7f4b7e0fb640 20 client.4697.objecter shutdown clearing up homeless session... 2024-03-25T18:18:36.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:36.099+0000 7f4b7e0fb640 10 client.4697.objecter successfully canceled tick 2024-03-25T18:18:36.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:36.099+0000 7f4b7e0fb640 1 -- 172.21.15.156:0/3275500876 >> v1:172.21.15.194:6800/2946427713 conn(0x7f4b6406ddf0 legacy=0x7f4b640702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:36.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:36.099+0000 7f4b7e0fb640 10 monclient: shutdown 2024-03-25T18:18:36.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:36.099+0000 7f4b7e0fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:36.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:36.099+0000 7f4b7e0fb640 1 -- 172.21.15.156:0/3275500876 >> v1:172.21.15.156:6790/0 conn(0x7f4b7813f710 legacy=0x7f4b781d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:36.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:36.099+0000 7f4b7e0fb640 1 -- 172.21.15.156:0/3275500876 shutdown_connections 2024-03-25T18:18:36.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:36.099+0000 7f4b7e0fb640 1 -- 172.21.15.156:0/3275500876 >> 172.21.15.156:0/3275500876 conn(0x7f4b780b7ea0 msgr2=0x7f4b7813d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:36.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:36.100+0000 7f4b7e0fb640 1 -- 172.21.15.156:0/3275500876 shutdown_connections 2024-03-25T18:18:36.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:36.100+0000 7f4b7e0fb640 1 -- 172.21.15.156:0/3275500876 wait complete. 2024-03-25T18:18:36.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:36.100+0000 7f4b7e0fb640 1 librados: shutdown 2024-03-25T18:18:36.113 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:40.186 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:10:46.176 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:40.193 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:10:46.183 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:40.224 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:10:46.214 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:40.243 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:10:46.233 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:40.246 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:10:46.212 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:40.251 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:10:46.218 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:40.251 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:10:46.242 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:40.284 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:10:46.251 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:40.295 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:10:46.286 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:42.920 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:43.035 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.034+0000 7f2ebf896640 10 monclient: get_monmap_and_config 2024-03-25T18:18:43.035 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.034+0000 7f2ebf896640 10 monclient: build_initial_monmap 2024-03-25T18:18:43.035 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.034+0000 7f2ebf896640 10 monclient: monmap: 2024-03-25T18:18:43.035 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:18:43.035 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:43.035 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:18:43.035306+0000 2024-03-25T18:18:43.036 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:18:43.035306+0000 2024-03-25T18:18:43.036 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:43.036 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:43.036 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:18:43.036 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:18:43.036 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:18:43.036 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:43.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.034+0000 7f2ebf896640 1 Processor -- start 2024-03-25T18:18:43.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.035+0000 7f2ebf896640 1 -- start start 2024-03-25T18:18:43.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.035+0000 7f2ebf896640 10 monclient: init 2024-03-25T18:18:43.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.035+0000 7f2ebf896640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:43.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.035+0000 7f2ebf896640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:43.038 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.035+0000 7f2ebf896640 10 monclient(hunting): picked mon.noname-a con 0x7f2eb80b4d60 addr v1:172.21.15.156:6789/0 2024-03-25T18:18:43.038 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.035+0000 7f2ebf896640 10 monclient(hunting): picked mon.noname-b con 0x7f2eb80b1730 addr v1:172.21.15.156:6790/0 2024-03-25T18:18:43.038 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.035+0000 7f2ebf896640 10 monclient(hunting): picked mon.noname-c con 0x7f2eb814be90 addr v1:172.21.15.194:6789/0 2024-03-25T18:18:43.038 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.035+0000 7f2ebf896640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb80af940 con 0x7f2eb80b4d60 2024-03-25T18:18:43.038 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.035+0000 7f2ebf896640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb80b0a70 con 0x7f2eb814be90 2024-03-25T18:18:43.038 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.035+0000 7f2ebf896640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb80b0d50 con 0x7f2eb80b1730 2024-03-25T18:18:43.038 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.035+0000 7f2ebf896640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:43.038 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.035+0000 7f2ebf896640 10 monclient(hunting): authenticate will time out at 7604.346726s 2024-03-25T18:18:43.045 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.044+0000 7f2ebde33640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f2eb814be90 0x7f2eb814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:43958/0 (socket says 172.21.15.156:43958) 2024-03-25T18:18:43.045 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.044+0000 7f2ebde33640 1 -- 172.21.15.156:0/3859188425 learned_addr learned my addr 172.21.15.156:0/3859188425 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:18:43.056 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.055+0000 7f2eaffff640 1 -- 172.21.15.156:0/3859188425 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3955142396 0 0) 0x7f2eb80b0a70 con 0x7f2eb814be90 2024-03-25T18:18:43.056 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.055+0000 7f2eaffff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:43.056 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.055+0000 7f2eaffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:43.056 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.055+0000 7f2eaffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:43.056 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.055+0000 7f2eaffff640 10 monclient(hunting): my global_id is 4696 2024-03-25T18:18:43.056 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.055+0000 7f2eaffff640 1 -- 172.21.15.156:0/3859188425 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ea4003520 con 0x7f2eb814be90 2024-03-25T18:18:43.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.075+0000 7f2eaffff640 1 -- 172.21.15.156:0/3859188425 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816007205 0 0) 0x7f2eb80b0d50 con 0x7f2eb80b1730 2024-03-25T18:18:43.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.075+0000 7f2eaffff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:43.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.075+0000 7f2eaffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:43.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.075+0000 7f2eaffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:43.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.075+0000 7f2eaffff640 10 monclient(hunting): my global_id is 4700 2024-03-25T18:18:43.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.075+0000 7f2eaffff640 1 -- 172.21.15.156:0/3859188425 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eb80b0a70 con 0x7f2eb80b1730 2024-03-25T18:18:43.086 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.085+0000 7f2eaffff640 1 -- 172.21.15.156:0/3859188425 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2427860683 0 0) 0x7f2eb80af940 con 0x7f2eb80b4d60 2024-03-25T18:18:43.086 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.085+0000 7f2eaffff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:43.086 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.085+0000 7f2eaffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:43.086 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.085+0000 7f2eaffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:43.086 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.085+0000 7f2eaffff640 10 monclient(hunting): my global_id is 4692 2024-03-25T18:18:43.086 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.085+0000 7f2eaffff640 1 -- 172.21.15.156:0/3859188425 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eb80b0d50 con 0x7f2eb80b4d60 2024-03-25T18:18:43.087 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.086+0000 7f2eaffff640 1 -- 172.21.15.156:0/3859188425 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3275194208 0 0) 0x7f2ea4003520 con 0x7f2eb814be90 2024-03-25T18:18:43.087 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.086+0000 7f2eaffff640 1 -- 172.21.15.156:0/3859188425 >> v1:172.21.15.156:6790/0 conn(0x7f2eb80b1730 legacy=0x7f2eb80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:43.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.086+0000 7f2eaffff640 1 -- 172.21.15.156:0/3859188425 >> v1:172.21.15.156:6789/0 conn(0x7f2eb80b4d60 legacy=0x7f2eb813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:43.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.086+0000 7f2eaffff640 10 monclient: _finish_hunting 0 2024-03-25T18:18:43.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.086+0000 7f2eaffff640 1 monclient: found mon.noname-c 2024-03-25T18:18:43.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.086+0000 7f2eaffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:43.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.086+0000 7f2eaffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:18:43.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.086+0000 7f2eaffff640 1 -- 172.21.15.156:0/3859188425 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb80b1030 con 0x7f2eb814be90 2024-03-25T18:18:43.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.086+0000 7f2eaffff640 10 monclient: _finish_auth 0 2024-03-25T18:18:43.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.086+0000 7f2eaffff640 10 monclient: _check_auth_tickets 2024-03-25T18:18:43.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.086+0000 7f2eaffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:43.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.086+0000 7f2eaffff640 1 -- 172.21.15.156:0/3859188425 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f2eb0001a10 con 0x7f2eb814be90 2024-03-25T18:18:43.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.086+0000 7f2eaffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:43.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.086+0000 7f2eaffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:43.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.086+0000 7f2eaffff640 10 monclient: dump: 2024-03-25T18:18:43.088 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:43.089 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:43.089 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:43.089 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:43.089 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:43.089 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:43.089 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:43.089 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:43.089 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:43.089 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:43.089 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.086+0000 7f2ebf896640 5 monclient: authenticate success, global_id 4696 2024-03-25T18:18:43.089 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.086+0000 7f2ebf896640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:43.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.113+0000 7f2eaffff640 1 -- 172.21.15.156:0/3859188425 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2eb0001b70 con 0x7f2eb814be90 2024-03-25T18:18:43.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.113+0000 7f2eaffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:43.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.113+0000 7f2eaffff640 1 -- 172.21.15.156:0/3859188425 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f2eb0001f80 con 0x7f2eb814be90 2024-03-25T18:18:43.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.113+0000 7f2eaffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:43.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.113+0000 7f2eaffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:43.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.113+0000 7f2eaffff640 10 monclient: dump: 2024-03-25T18:18:43.115 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:43.115 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:43.115 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:43.115 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:43.115 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:43.115 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:43.115 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:43.116 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:43.116 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:43.116 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:43.116 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.114+0000 7f2ebf896640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:43.116 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.114+0000 7f2ebf896640 4 set_mon_vals no callback set 2024-03-25T18:18:43.117 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.115+0000 7f2ebf896640 10 monclient: shutdown 2024-03-25T18:18:43.117 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.115+0000 7f2ebf896640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:43.117 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.115+0000 7f2ebf896640 1 -- 172.21.15.156:0/3859188425 >> v1:172.21.15.194:6789/0 conn(0x7f2eb814be90 legacy=0x7f2eb814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:43.117 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.115+0000 7f2ebf896640 1 -- 172.21.15.156:0/3859188425 shutdown_connections 2024-03-25T18:18:43.117 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.115+0000 7f2ebf896640 1 -- 172.21.15.156:0/3859188425 >> 172.21.15.156:0/3859188425 conn(0x7f2eb80617a0 msgr2=0x7f2eb8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:43.117 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.115+0000 7f2ebf896640 1 -- 172.21.15.156:0/3859188425 shutdown_connections 2024-03-25T18:18:43.118 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.115+0000 7f2ebf896640 1 -- 172.21.15.156:0/3859188425 wait complete. 2024-03-25T18:18:43.118 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.116+0000 7f2ebf896640 10 monclient: build_initial_monmap 2024-03-25T18:18:43.118 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.116+0000 7f2ebf896640 10 monclient: monmap: 2024-03-25T18:18:43.118 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:18:43.118 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:43.118 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:18:43.118 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:18:43.118 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:43.118 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:18:43.118 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:18:43.118 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:18:43.118 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:18:43.118 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:43.118 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.116+0000 7f2ebf896640 1 librados: starting msgr at 2024-03-25T18:18:43.118 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.116+0000 7f2ebf896640 1 librados: starting objecter 2024-03-25T18:18:43.119 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.116+0000 7f2ebf896640 1 Processor -- start 2024-03-25T18:18:43.119 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.116+0000 7f2ebf896640 1 -- start start 2024-03-25T18:18:43.119 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.116+0000 7f2ebf896640 1 librados: setting wanted keys 2024-03-25T18:18:43.119 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.116+0000 7f2ebf896640 1 librados: calling monclient init 2024-03-25T18:18:43.119 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.116+0000 7f2ebf896640 10 monclient: init 2024-03-25T18:18:43.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.117+0000 7f2ebf896640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:43.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.117+0000 7f2ebf896640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:43.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.117+0000 7f2ebf896640 10 monclient(hunting): picked mon.noname-a con 0x7f2eb80b1730 addr v1:172.21.15.156:6789/0 2024-03-25T18:18:43.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.117+0000 7f2ebf896640 10 monclient(hunting): picked mon.noname-c con 0x7f2eb80b4d60 addr v1:172.21.15.156:6790/0 2024-03-25T18:18:43.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.117+0000 7f2ebf896640 10 monclient(hunting): picked mon.noname-b con 0x7f2eb814be90 addr v1:172.21.15.194:6789/0 2024-03-25T18:18:43.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.117+0000 7f2ebf896640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb81363f0 con 0x7f2eb80b1730 2024-03-25T18:18:43.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.117+0000 7f2ebf896640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb81374e0 con 0x7f2eb814be90 2024-03-25T18:18:43.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.117+0000 7f2ebf896640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb8138140 con 0x7f2eb80b4d60 2024-03-25T18:18:43.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.117+0000 7f2ebf896640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:43.120 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.117+0000 7f2ebf896640 10 monclient(hunting): authenticate will time out at 7604.428239s 2024-03-25T18:18:43.127 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.126+0000 7f2ebd632640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f2eb80b1730 0x7f2eb8136dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:51452/0 (socket says 172.21.15.156:51452) 2024-03-25T18:18:43.127 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.126+0000 7f2ebd632640 1 -- 172.21.15.156:0/2183243246 learned_addr learned my addr 172.21.15.156:0/2183243246 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:18:43.131 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.130+0000 7f2eadffb640 10 client.?.objecter ms_handle_connect 0x7f2eb80b1730 2024-03-25T18:18:43.132 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.130+0000 7f2eadffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:43.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.133+0000 7f2eadffb640 10 client.?.objecter ms_handle_connect 0x7f2eb814be90 2024-03-25T18:18:43.134 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.133+0000 7f2eadffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:43.137 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.136+0000 7f2eadffb640 1 -- 172.21.15.156:0/2183243246 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3825012639 0 0) 0x7f2eb81363f0 con 0x7f2eb80b1730 2024-03-25T18:18:43.137 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.136+0000 7f2eadffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:43.137 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.136+0000 7f2eadffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:43.137 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.136+0000 7f2eadffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:43.137 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.136+0000 7f2eadffb640 10 monclient(hunting): my global_id is 4695 2024-03-25T18:18:43.137 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.136+0000 7f2eadffb640 1 -- 172.21.15.156:0/2183243246 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e9c004eb0 con 0x7f2eb80b1730 2024-03-25T18:18:43.139 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.138+0000 7f2eadffb640 10 client.?.objecter ms_handle_connect 0x7f2eb80b4d60 2024-03-25T18:18:43.139 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.138+0000 7f2eadffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:43.142 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.141+0000 7f2eadffb640 1 -- 172.21.15.156:0/2183243246 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2275677456 0 0) 0x7f2eb81374e0 con 0x7f2eb814be90 2024-03-25T18:18:43.142 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.141+0000 7f2eadffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:43.142 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.141+0000 7f2eadffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:43.142 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.141+0000 7f2eadffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:43.142 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.141+0000 7f2eadffb640 10 monclient(hunting): my global_id is 4699 2024-03-25T18:18:43.142 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.141+0000 7f2eadffb640 1 -- 172.21.15.156:0/2183243246 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eb81363f0 con 0x7f2eb814be90 2024-03-25T18:18:43.149 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.148+0000 7f2eadffb640 1 -- 172.21.15.156:0/2183243246 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461633632 0 0) 0x7f2eb8138140 con 0x7f2eb80b4d60 2024-03-25T18:18:43.149 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.148+0000 7f2eadffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:43.149 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.148+0000 7f2eadffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:43.149 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.148+0000 7f2eadffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:43.149 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.148+0000 7f2eadffb640 10 monclient(hunting): my global_id is 4703 2024-03-25T18:18:43.149 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.148+0000 7f2eadffb640 1 -- 172.21.15.156:0/2183243246 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eb81374e0 con 0x7f2eb80b4d60 2024-03-25T18:18:43.155 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.154+0000 7f2eadffb640 1 -- 172.21.15.156:0/2183243246 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2654620411 0 0) 0x7f2e9c004eb0 con 0x7f2eb80b1730 2024-03-25T18:18:43.156 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.154+0000 7f2eadffb640 1 -- 172.21.15.156:0/2183243246 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2eb8138140 con 0x7f2eb80b1730 2024-03-25T18:18:43.156 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.154+0000 7f2eadffb640 1 -- 172.21.15.156:0/2183243246 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f2eb4001a10 con 0x7f2eb80b1730 2024-03-25T18:18:43.156 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.154+0000 7f2eadffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:43.156 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.154+0000 7f2eadffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:43.156 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.154+0000 7f2eadffb640 10 monclient(hunting): dump: 2024-03-25T18:18:43.156 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:43.156 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:43.156 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:43.156 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:43.156 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:43.156 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:43.156 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:43.156 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:43.156 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:43.156 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:43.157 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.154+0000 7f2eadffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:43.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.157+0000 7f2eadffb640 1 -- 172.21.15.156:0/2183243246 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 153577201 0 0) 0x7f2eb81363f0 con 0x7f2eb814be90 2024-03-25T18:18:43.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.157+0000 7f2eadffb640 1 -- 172.21.15.156:0/2183243246 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e9c004eb0 con 0x7f2eb814be90 2024-03-25T18:18:43.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.157+0000 7f2eadffb640 1 -- 172.21.15.156:0/2183243246 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f2eb0001e80 con 0x7f2eb814be90 2024-03-25T18:18:43.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.157+0000 7f2eadffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:43.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.157+0000 7f2eadffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:43.158 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.157+0000 7f2eadffb640 10 monclient(hunting): dump: 2024-03-25T18:18:43.158 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:43.158 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:43.158 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:43.158 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:43.159 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:43.159 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:43.159 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:43.159 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:43.159 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:43.159 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:43.166 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.165+0000 7f2eadffb640 1 -- 172.21.15.156:0/2183243246 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2875918799 0 0) 0x7f2e9c004eb0 con 0x7f2eb814be90 2024-03-25T18:18:43.166 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.165+0000 7f2eadffb640 1 -- 172.21.15.156:0/2183243246 >> v1:172.21.15.156:6790/0 conn(0x7f2eb80b4d60 legacy=0x7f2eb812fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:43.166 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.165+0000 7f2eadffb640 1 -- 172.21.15.156:0/2183243246 >> v1:172.21.15.156:6789/0 conn(0x7f2eb80b1730 legacy=0x7f2eb8136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:43.166 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.165+0000 7f2eadffb640 10 monclient: _finish_hunting 0 2024-03-25T18:18:43.166 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.165+0000 7f2eadffb640 1 monclient: found mon.b 2024-03-25T18:18:43.166 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.165+0000 7f2eadffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:43.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.165+0000 7f2eadffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:18:43.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.165+0000 7f2eadffb640 1 -- 172.21.15.156:0/2183243246 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb8139090 con 0x7f2eb814be90 2024-03-25T18:18:43.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.165+0000 7f2eadffb640 10 monclient: _finish_auth 0 2024-03-25T18:18:43.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.165+0000 7f2eadffb640 10 monclient: _check_auth_tickets 2024-03-25T18:18:43.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.165+0000 7f2eadffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:43.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.165+0000 7f2ebf896640 5 monclient: authenticate success, global_id 4699 2024-03-25T18:18:43.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.165+0000 7f2ebf896640 10 monclient: _renew_subs 2024-03-25T18:18:43.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.165+0000 7f2ebf896640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:18:43.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.165+0000 7f2ebf896640 1 -- 172.21.15.156:0/2183243246 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb8136030 con 0x7f2eb814be90 2024-03-25T18:18:43.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.165+0000 7f2ebf896640 10 client.4699.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:43.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.165+0000 7f2ebf896640 10 monclient: _renew_subs 2024-03-25T18:18:43.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.165+0000 7f2ebf896640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:18:43.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.165+0000 7f2ebf896640 1 -- 172.21.15.156:0/2183243246 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb8130650 con 0x7f2eb814be90 2024-03-25T18:18:43.167 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.165+0000 7f2ebf896640 1 librados: init done 2024-03-25T18:18:43.168 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.166+0000 7f2ebf896640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:43.168 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.166+0000 7f2ebf896640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:43.168 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.166+0000 7f2ebf896640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:18:43.168 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.166+0000 7f2ebf896640 1 -- 172.21.15.156:0/2183243246 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eb80ae6c0 con 0x7f2eb814be90 2024-03-25T18:18:43.438 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.437+0000 7f2eadffb640 1 -- 172.21.15.156:0/2183243246 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2eb0001fe0 con 0x7f2eb814be90 2024-03-25T18:18:43.438 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.437+0000 7f2eadffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:43.439 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.437+0000 7f2eadffb640 1 -- 172.21.15.156:0/2183243246 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f2eb0004fb0 con 0x7f2eb814be90 2024-03-25T18:18:43.439 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.437+0000 7f2eadffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:43.439 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.437+0000 7f2eadffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:43.439 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.437+0000 7f2eadffb640 10 monclient: dump: 2024-03-25T18:18:43.439 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:43.439 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:43.439 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:43.439 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:43.439 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:43.439 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:43.439 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:43.439 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:43.439 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:43.440 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:43.440 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.437+0000 7f2eaf7fe640 4 set_mon_vals no callback set 2024-03-25T18:18:43.440 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.438+0000 7f2eadffb640 1 -- 172.21.15.156:0/2183243246 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f2eb001af80 con 0x7f2eb814be90 2024-03-25T18:18:43.440 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.439+0000 7f2ebd632640 10 monclient: get_auth_request con 0x7f2e9c06df70 auth_method 0 2024-03-25T18:18:43.440 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.439+0000 7f2eadffb640 10 client.4699.objecter ms_handle_connect 0x7f2e9c06df70 2024-03-25T18:18:43.444 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.443+0000 7f2eadffb640 1 -- 172.21.15.156:0/2183243246 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3381691338 0 0) 0x7f2eb0054a50 con 0x7f2eb814be90 2024-03-25T18:18:43.444 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.443+0000 7f2eadffb640 10 client.4699.objecter ms_dispatch 0x7f2eb8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:18:43.444 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.443+0000 7f2eadffb640 3 client.4699.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:18:43.444 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.443+0000 7f2eadffb640 3 client.4699.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:18:43.444 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.443+0000 7f2eadffb640 20 client.4699.objecter dump_active .. 0 homeless 2024-03-25T18:18:43.444 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.443+0000 7f2eadffb640 1 -- 172.21.15.156:0/2183243246 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2eb0050560 con 0x7f2eb814be90 2024-03-25T18:18:43.445 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.443+0000 7f2eadffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:43.445 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.443+0000 7f2eadffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:43.692 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.691+0000 7f2ebf896640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:43.692 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.691+0000 7f2ebf896640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:43.692 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.691+0000 7f2ebf896640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:18:43.693 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.691+0000 7f2ebf896640 1 -- 172.21.15.156:0/2183243246 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2eb80627b0 con 0x7f2eb814be90 2024-03-25T18:18:43.712 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.711+0000 7f2eadffb640 1 -- 172.21.15.156:0/2183243246 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1441834839) 0x7f2eb0052490 con 0x7f2eb814be90 2024-03-25T18:18:43.713 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.711+0000 7f2eadffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:43.713 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.711+0000 7f2eadffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:43.713 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:18:43.713 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":11,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:18:20.029906+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:18:20.029906+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:43.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.714+0000 7f2ebf896640 10 librados: watch_flush enter 2024-03-25T18:18:43.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.714+0000 7f2ebf896640 10 librados: watch_flush exit 2024-03-25T18:18:43.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.714+0000 7f2ebf896640 20 client.4699.objecter shutdown clearing up homeless session... 2024-03-25T18:18:43.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.714+0000 7f2ebf896640 10 client.4699.objecter successfully canceled tick 2024-03-25T18:18:43.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.714+0000 7f2ebf896640 1 -- 172.21.15.156:0/2183243246 >> v1:172.21.15.194:6800/2946427713 conn(0x7f2e9c06df70 legacy=0x7f2e9c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:43.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.714+0000 7f2ebf896640 10 monclient: shutdown 2024-03-25T18:18:43.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.714+0000 7f2ebf896640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:43.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.714+0000 7f2ebf896640 1 -- 172.21.15.156:0/2183243246 >> v1:172.21.15.194:6789/0 conn(0x7f2eb814be90 legacy=0x7f2eb8143c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:43.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.714+0000 7f2ebf896640 1 -- 172.21.15.156:0/2183243246 shutdown_connections 2024-03-25T18:18:43.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.714+0000 7f2ebf896640 1 -- 172.21.15.156:0/2183243246 >> 172.21.15.156:0/2183243246 conn(0x7f2eb80617a0 msgr2=0x7f2eb813c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:43.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.714+0000 7f2ebf896640 1 -- 172.21.15.156:0/2183243246 shutdown_connections 2024-03-25T18:18:43.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.714+0000 7f2ebf896640 1 -- 172.21.15.156:0/2183243246 wait complete. 2024-03-25T18:18:43.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:43.714+0000 7f2ebf896640 1 librados: shutdown 2024-03-25T18:18:43.727 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:50.532 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:50.649 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.648+0000 7f9ffb281640 10 monclient: get_monmap_and_config 2024-03-25T18:18:50.650 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.648+0000 7f9ffb281640 10 monclient: build_initial_monmap 2024-03-25T18:18:50.650 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.648+0000 7f9ffb281640 10 monclient: monmap: 2024-03-25T18:18:50.650 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:18:50.650 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:50.650 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:18:50.649875+0000 2024-03-25T18:18:50.650 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:18:50.649875+0000 2024-03-25T18:18:50.650 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:50.650 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:50.650 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:18:50.650 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:18:50.650 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:18:50.650 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:50.651 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.649+0000 7f9ffb281640 1 Processor -- start 2024-03-25T18:18:50.652 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.649+0000 7f9ffb281640 1 -- start start 2024-03-25T18:18:50.652 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.649+0000 7f9ffb281640 10 monclient: init 2024-03-25T18:18:50.652 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.650+0000 7f9ffb281640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:50.652 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.650+0000 7f9ffb281640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:18:50.652 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.650+0000 7f9ffb281640 10 monclient(hunting): picked mon.noname-c con 0x7f9ff40b4d60 addr v1:172.21.15.194:6789/0 2024-03-25T18:18:50.652 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.650+0000 7f9ffb281640 10 monclient(hunting): picked mon.noname-b con 0x7f9ff40b1730 addr v1:172.21.15.156:6790/0 2024-03-25T18:18:50.652 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.650+0000 7f9ffb281640 10 monclient(hunting): picked mon.noname-a con 0x7f9ff414be90 addr v1:172.21.15.156:6789/0 2024-03-25T18:18:50.652 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.650+0000 7f9ffb281640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ff40af940 con 0x7f9ff414be90 2024-03-25T18:18:50.652 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.650+0000 7f9ffb281640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ff40b0a70 con 0x7f9ff40b4d60 2024-03-25T18:18:50.652 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.650+0000 7f9ffb281640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ff40b0d50 con 0x7f9ff40b1730 2024-03-25T18:18:50.652 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.650+0000 7f9ffb281640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:50.652 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.650+0000 7f9ffb281640 10 monclient(hunting): authenticate will time out at 7611.961133s 2024-03-25T18:18:50.658 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.657+0000 7f9ff881c640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f9ff40b1730 0x7f9ff40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:44054/0 (socket says 172.21.15.156:44054) 2024-03-25T18:18:50.658 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.657+0000 7f9ff881c640 1 -- 172.21.15.156:0/4273015880 learned_addr learned my addr 172.21.15.156:0/4273015880 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:18:50.668 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.667+0000 7f9febfff640 1 -- 172.21.15.156:0/4273015880 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 977865848 0 0) 0x7f9ff40b0a70 con 0x7f9ff40b4d60 2024-03-25T18:18:50.668 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.667+0000 7f9febfff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:50.668 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.667+0000 7f9febfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:50.668 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.667+0000 7f9febfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:50.668 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.667+0000 7f9febfff640 10 monclient(hunting): my global_id is 4702 2024-03-25T18:18:50.668 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.667+0000 7f9febfff640 1 -- 172.21.15.156:0/4273015880 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fe0003520 con 0x7f9ff40b4d60 2024-03-25T18:18:50.669 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.667+0000 7f9febfff640 1 -- 172.21.15.156:0/4273015880 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3127404548 0 0) 0x7f9ff40b0d50 con 0x7f9ff40b1730 2024-03-25T18:18:50.669 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.667+0000 7f9febfff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:50.669 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.667+0000 7f9febfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:50.669 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.667+0000 7f9febfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:50.669 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.667+0000 7f9febfff640 10 monclient(hunting): my global_id is 4706 2024-03-25T18:18:50.669 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.667+0000 7f9febfff640 1 -- 172.21.15.156:0/4273015880 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ff40b0a70 con 0x7f9ff40b1730 2024-03-25T18:18:50.676 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.675+0000 7f9febfff640 1 -- 172.21.15.156:0/4273015880 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1934452678 0 0) 0x7f9fe0003520 con 0x7f9ff40b4d60 2024-03-25T18:18:50.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.675+0000 7f9febfff640 1 -- 172.21.15.156:0/4273015880 >> v1:172.21.15.156:6790/0 conn(0x7f9ff40b1730 legacy=0x7f9ff40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:50.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.675+0000 7f9febfff640 1 -- 172.21.15.156:0/4273015880 >> v1:172.21.15.156:6789/0 conn(0x7f9ff414be90 legacy=0x7f9ff414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:18:50.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.675+0000 7f9febfff640 10 monclient: _finish_hunting 0 2024-03-25T18:18:50.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.675+0000 7f9febfff640 1 monclient: found mon.noname-c 2024-03-25T18:18:50.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.675+0000 7f9febfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:50.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.675+0000 7f9febfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:18:50.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.675+0000 7f9febfff640 1 -- 172.21.15.156:0/4273015880 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff40b1030 con 0x7f9ff40b4d60 2024-03-25T18:18:50.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.675+0000 7f9febfff640 10 monclient: _finish_auth 0 2024-03-25T18:18:50.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.675+0000 7f9febfff640 10 monclient: _check_auth_tickets 2024-03-25T18:18:50.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.675+0000 7f9febfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:50.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.675+0000 7f9febfff640 1 -- 172.21.15.156:0/4273015880 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9ff0001690 con 0x7f9ff40b4d60 2024-03-25T18:18:50.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.675+0000 7f9febfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:50.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.675+0000 7f9febfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:50.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.675+0000 7f9febfff640 10 monclient: dump: 2024-03-25T18:18:50.677 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:50.678 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:50.678 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:50.678 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:50.678 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:50.678 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:50.678 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:50.678 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:50.678 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:50.678 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:50.678 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.675+0000 7f9ffb281640 5 monclient: authenticate success, global_id 4702 2024-03-25T18:18:50.678 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.675+0000 7f9ffb281640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:50.691 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.690+0000 7f9febfff640 1 -- 172.21.15.156:0/4273015880 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ff00017f0 con 0x7f9ff40b4d60 2024-03-25T18:18:50.691 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.690+0000 7f9febfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:50.691 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.690+0000 7f9febfff640 1 -- 172.21.15.156:0/4273015880 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9ff0001c00 con 0x7f9ff40b4d60 2024-03-25T18:18:50.691 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.690+0000 7f9febfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:50.692 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.690+0000 7f9febfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:50.692 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.690+0000 7f9febfff640 10 monclient: dump: 2024-03-25T18:18:50.692 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:50.692 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:50.692 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:50.692 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:50.692 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:50.692 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:50.692 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:50.692 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:50.692 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:50.692 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:50.692 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.690+0000 7f9ffb281640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:50.692 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.690+0000 7f9ffb281640 4 set_mon_vals no callback set 2024-03-25T18:18:50.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.691+0000 7f9ffb281640 10 monclient: shutdown 2024-03-25T18:18:50.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.691+0000 7f9ffb281640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:50.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.691+0000 7f9ffb281640 1 -- 172.21.15.156:0/4273015880 >> v1:172.21.15.194:6789/0 conn(0x7f9ff40b4d60 legacy=0x7f9ff413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:50.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.691+0000 7f9ffb281640 1 -- 172.21.15.156:0/4273015880 shutdown_connections 2024-03-25T18:18:50.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.691+0000 7f9ffb281640 1 -- 172.21.15.156:0/4273015880 >> 172.21.15.156:0/4273015880 conn(0x7f9ff40617a0 msgr2=0x7f9ff4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:50.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.691+0000 7f9ffb281640 1 -- 172.21.15.156:0/4273015880 shutdown_connections 2024-03-25T18:18:50.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.691+0000 7f9ffb281640 1 -- 172.21.15.156:0/4273015880 wait complete. 2024-03-25T18:18:50.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.692+0000 7f9ffb281640 10 monclient: build_initial_monmap 2024-03-25T18:18:50.694 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.692+0000 7f9ffb281640 10 monclient: monmap: 2024-03-25T18:18:50.694 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:18:50.695 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:50.695 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:18:50.695 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:18:50.695 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:50.695 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:18:50.695 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:18:50.695 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:18:50.695 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:18:50.695 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:50.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.692+0000 7f9ffb281640 1 librados: starting msgr at 2024-03-25T18:18:50.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.692+0000 7f9ffb281640 1 librados: starting objecter 2024-03-25T18:18:50.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.693+0000 7f9ffb281640 1 Processor -- start 2024-03-25T18:18:50.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.693+0000 7f9ffb281640 1 -- start start 2024-03-25T18:18:50.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.693+0000 7f9ffb281640 1 librados: setting wanted keys 2024-03-25T18:18:50.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.693+0000 7f9ffb281640 1 librados: calling monclient init 2024-03-25T18:18:50.696 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.693+0000 7f9ffb281640 10 monclient: init 2024-03-25T18:18:50.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.693+0000 7f9ffb281640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:50.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.693+0000 7f9ffb281640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:18:50.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.693+0000 7f9ffb281640 10 monclient(hunting): picked mon.noname-b con 0x7f9ff40b1730 addr v1:172.21.15.194:6789/0 2024-03-25T18:18:50.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.693+0000 7f9ffb281640 10 monclient(hunting): picked mon.noname-a con 0x7f9ff40b4d60 addr v1:172.21.15.156:6789/0 2024-03-25T18:18:50.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.693+0000 7f9ffb281640 10 monclient(hunting): picked mon.noname-c con 0x7f9ff414be90 addr v1:172.21.15.156:6790/0 2024-03-25T18:18:50.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.693+0000 7f9ffb281640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ff41d7850 con 0x7f9ff40b4d60 2024-03-25T18:18:50.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.693+0000 7f9ffb281640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ff413a3d0 con 0x7f9ff40b1730 2024-03-25T18:18:50.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.693+0000 7f9ffb281640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ff413bb10 con 0x7f9ff414be90 2024-03-25T18:18:50.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.693+0000 7f9ffb281640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:50.697 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.693+0000 7f9ffb281640 10 monclient(hunting): authenticate will time out at 7612.004687s 2024-03-25T18:18:50.705 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.704+0000 7f9ff981e640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f9ff414be90 0x7f9ff41e2060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:44106/0 (socket says 172.21.15.156:44106) 2024-03-25T18:18:50.705 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.704+0000 7f9ff981e640 1 -- 172.21.15.156:0/3714013163 learned_addr learned my addr 172.21.15.156:0/3714013163 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:18:50.706 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.705+0000 7f9fe9ffb640 10 client.?.objecter ms_handle_connect 0x7f9ff414be90 2024-03-25T18:18:50.706 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.705+0000 7f9fe9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:50.708 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.707+0000 7f9fe9ffb640 10 client.?.objecter ms_handle_connect 0x7f9ff40b1730 2024-03-25T18:18:50.708 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.707+0000 7f9fe9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:50.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.714+0000 7f9fe9ffb640 10 client.?.objecter ms_handle_connect 0x7f9ff40b4d60 2024-03-25T18:18:50.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.714+0000 7f9fe9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:50.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.715+0000 7f9fe9ffb640 1 -- 172.21.15.156:0/3714013163 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1301453509 0 0) 0x7f9ff413a3d0 con 0x7f9ff40b1730 2024-03-25T18:18:50.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.715+0000 7f9fe9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:50.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.715+0000 7f9fe9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:50.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.715+0000 7f9fe9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:50.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.715+0000 7f9fe9ffb640 10 monclient(hunting): my global_id is 4705 2024-03-25T18:18:50.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.715+0000 7f9fe9ffb640 1 -- 172.21.15.156:0/3714013163 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fd8004eb0 con 0x7f9ff40b1730 2024-03-25T18:18:50.720 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.719+0000 7f9fe9ffb640 1 -- 172.21.15.156:0/3714013163 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1970000611 0 0) 0x7f9ff413bb10 con 0x7f9ff414be90 2024-03-25T18:18:50.720 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.719+0000 7f9fe9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:50.720 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.719+0000 7f9fe9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:50.720 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.719+0000 7f9fe9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:50.720 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.719+0000 7f9fe9ffb640 10 monclient(hunting): my global_id is 4709 2024-03-25T18:18:50.720 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.719+0000 7f9fe9ffb640 1 -- 172.21.15.156:0/3714013163 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ff413a3d0 con 0x7f9ff414be90 2024-03-25T18:18:50.722 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.721+0000 7f9fe9ffb640 1 -- 172.21.15.156:0/3714013163 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3955779768 0 0) 0x7f9fd8004eb0 con 0x7f9ff40b1730 2024-03-25T18:18:50.722 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.721+0000 7f9fe9ffb640 1 -- 172.21.15.156:0/3714013163 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ff413bb10 con 0x7f9ff40b1730 2024-03-25T18:18:50.722 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.721+0000 7f9fe9ffb640 1 -- 172.21.15.156:0/3714013163 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9ff0001b40 con 0x7f9ff40b1730 2024-03-25T18:18:50.722 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.721+0000 7f9fe9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:50.722 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.721+0000 7f9fe9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:50.722 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.721+0000 7f9fe9ffb640 10 monclient(hunting): dump: 2024-03-25T18:18:50.722 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:50.722 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:50.723 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:50.723 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:50.723 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:50.723 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:50.723 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:50.723 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:50.723 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:50.723 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:50.723 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.721+0000 7f9fe9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:50.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.726+0000 7f9fe9ffb640 1 -- 172.21.15.156:0/3714013163 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2592807194 0 0) 0x7f9ff413bb10 con 0x7f9ff40b1730 2024-03-25T18:18:50.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.727+0000 7f9fe9ffb640 1 -- 172.21.15.156:0/3714013163 >> v1:172.21.15.156:6790/0 conn(0x7f9ff414be90 legacy=0x7f9ff41e2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:50.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.727+0000 7f9fe9ffb640 1 -- 172.21.15.156:0/3714013163 >> v1:172.21.15.156:6789/0 conn(0x7f9ff40b4d60 legacy=0x7f9ff41d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:50.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.727+0000 7f9fe9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:18:50.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.727+0000 7f9fe9ffb640 1 monclient: found mon.b 2024-03-25T18:18:50.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.727+0000 7f9fe9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:50.728 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.727+0000 7f9fe9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:18:50.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.727+0000 7f9fe9ffb640 1 -- 172.21.15.156:0/3714013163 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff41d83d0 con 0x7f9ff40b1730 2024-03-25T18:18:50.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.727+0000 7f9fe9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:18:50.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.727+0000 7f9fe9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:18:50.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.727+0000 7f9fe9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:50.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.727+0000 7f9ffb281640 5 monclient: authenticate success, global_id 4705 2024-03-25T18:18:50.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.727+0000 7f9ffb281640 10 monclient: _renew_subs 2024-03-25T18:18:50.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.727+0000 7f9ffb281640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:18:50.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.727+0000 7f9ffb281640 1 -- 172.21.15.156:0/3714013163 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ff4138d60 con 0x7f9ff40b1730 2024-03-25T18:18:50.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.727+0000 7f9ffb281640 10 client.4705.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:50.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.727+0000 7f9ffb281640 10 monclient: _renew_subs 2024-03-25T18:18:50.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.727+0000 7f9ffb281640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:18:50.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.727+0000 7f9ffb281640 1 -- 172.21.15.156:0/3714013163 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ff413d2d0 con 0x7f9ff40b1730 2024-03-25T18:18:50.729 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.727+0000 7f9ffb281640 1 librados: init done 2024-03-25T18:18:50.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.728+0000 7f9ffb281640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:50.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.728+0000 7f9ffb281640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:50.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.728+0000 7f9ffb281640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:18:50.730 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.728+0000 7f9ffb281640 1 -- 172.21.15.156:0/3714013163 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ff40627b0 con 0x7f9ff40b1730 2024-03-25T18:18:50.993 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.992+0000 7f9fe9ffb640 1 -- 172.21.15.156:0/3714013163 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ff00038a0 con 0x7f9ff40b1730 2024-03-25T18:18:50.993 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.992+0000 7f9fe9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:50.993 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.992+0000 7f9fe9ffb640 1 -- 172.21.15.156:0/3714013163 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f9ff0001250 con 0x7f9ff40b1730 2024-03-25T18:18:50.993 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.992+0000 7f9fe9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:50.993 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.992+0000 7f9fe9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:50.993 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.992+0000 7f9fe9ffb640 10 monclient: dump: 2024-03-25T18:18:50.993 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:50.993 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:50.993 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:50.994 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:50.994 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:50.994 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:50.994 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:50.994 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:50.994 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:50.994 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:50.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.992+0000 7f9feb7fe640 4 set_mon_vals no callback set 2024-03-25T18:18:50.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.992+0000 7f9fe9ffb640 1 -- 172.21.15.156:0/3714013163 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f9ff0019aa0 con 0x7f9ff40b1730 2024-03-25T18:18:50.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.993+0000 7f9ff881c640 10 monclient: get_auth_request con 0x7f9fd806dda0 auth_method 0 2024-03-25T18:18:50.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:50.993+0000 7f9fe9ffb640 10 client.4705.objecter ms_handle_connect 0x7f9fd806dda0 2024-03-25T18:18:51.002 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.001+0000 7f9fe9ffb640 1 -- 172.21.15.156:0/3714013163 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3381691338 0 0) 0x7f9ff0053730 con 0x7f9ff40b1730 2024-03-25T18:18:51.002 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.001+0000 7f9fe9ffb640 10 client.4705.objecter ms_dispatch 0x7f9ff4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:18:51.002 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.001+0000 7f9fe9ffb640 3 client.4705.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:18:51.002 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.001+0000 7f9fe9ffb640 3 client.4705.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:18:51.002 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.001+0000 7f9fe9ffb640 20 client.4705.objecter dump_active .. 0 homeless 2024-03-25T18:18:51.003 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.002+0000 7f9fe9ffb640 1 -- 172.21.15.156:0/3714013163 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9ff004f070 con 0x7f9ff40b1730 2024-03-25T18:18:51.003 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.002+0000 7f9fe9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:51.003 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.002+0000 7f9fe9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:51.257 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.256+0000 7f9ffb281640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:51.258 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.256+0000 7f9ffb281640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:51.258 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.256+0000 7f9ffb281640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:18:51.258 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.256+0000 7f9ffb281640 1 -- 172.21.15.156:0/3714013163 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ff4069f40 con 0x7f9ff40b1730 2024-03-25T18:18:51.287 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.286+0000 7f9fe9ffb640 1 -- 172.21.15.156:0/3714013163 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1441834839) 0x7f9ff0051170 con 0x7f9ff40b1730 2024-03-25T18:18:51.287 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.286+0000 7f9fe9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:51.287 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.286+0000 7f9fe9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:51.287 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:18:51.287 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":11,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:18:20.029906+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:18:20.029906+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:51.290 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.288+0000 7f9ffb281640 10 librados: watch_flush enter 2024-03-25T18:18:51.290 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.288+0000 7f9ffb281640 10 librados: watch_flush exit 2024-03-25T18:18:51.290 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.288+0000 7f9ffb281640 20 client.4705.objecter shutdown clearing up homeless session... 2024-03-25T18:18:51.290 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.288+0000 7f9ffb281640 10 client.4705.objecter successfully canceled tick 2024-03-25T18:18:51.290 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.288+0000 7f9ffb281640 1 -- 172.21.15.156:0/3714013163 >> v1:172.21.15.194:6800/2946427713 conn(0x7f9fd806dda0 legacy=0x7f9fd8070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:51.290 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.288+0000 7f9ffb281640 10 monclient: shutdown 2024-03-25T18:18:51.290 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.288+0000 7f9ffb281640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:51.290 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.288+0000 7f9ffb281640 1 -- 172.21.15.156:0/3714013163 >> v1:172.21.15.194:6789/0 conn(0x7f9ff40b1730 legacy=0x7f9ff4139cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:51.290 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.288+0000 7f9ffb281640 1 -- 172.21.15.156:0/3714013163 shutdown_connections 2024-03-25T18:18:51.290 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.288+0000 7f9ffb281640 1 -- 172.21.15.156:0/3714013163 >> 172.21.15.156:0/3714013163 conn(0x7f9ff40617a0 msgr2=0x7f9ff40b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:51.290 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.289+0000 7f9ffb281640 1 -- 172.21.15.156:0/3714013163 shutdown_connections 2024-03-25T18:18:51.290 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.289+0000 7f9ffb281640 1 -- 172.21.15.156:0/3714013163 wait complete. 2024-03-25T18:18:51.291 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:51.289+0000 7f9ffb281640 1 librados: shutdown 2024-03-25T18:18:51.302 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:58.108 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:58.221 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.220+0000 7f7fe859a640 10 monclient: get_monmap_and_config 2024-03-25T18:18:58.221 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.220+0000 7f7fe859a640 10 monclient: build_initial_monmap 2024-03-25T18:18:58.222 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.220+0000 7f7fe859a640 10 monclient: monmap: 2024-03-25T18:18:58.222 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:18:58.222 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:58.222 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:18:58.221744+0000 2024-03-25T18:18:58.222 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:18:58.221744+0000 2024-03-25T18:18:58.222 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:58.222 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:58.222 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:18:58.222 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:18:58.222 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:18:58.222 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:58.223 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.221+0000 7f7fe859a640 1 Processor -- start 2024-03-25T18:18:58.223 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.221+0000 7f7fe859a640 1 -- start start 2024-03-25T18:18:58.223 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.221+0000 7f7fe859a640 10 monclient: init 2024-03-25T18:18:58.224 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.222+0000 7f7fe859a640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:58.224 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.222+0000 7f7fe859a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:58.224 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.222+0000 7f7fe859a640 10 monclient(hunting): picked mon.noname-b con 0x7f7fe00a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:18:58.224 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.222+0000 7f7fe859a640 10 monclient(hunting): picked mon.noname-a con 0x7f7fe013f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:18:58.224 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.222+0000 7f7fe859a640 10 monclient(hunting): picked mon.noname-c con 0x7f7fe01430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:18:58.224 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.222+0000 7f7fe859a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe0142520 con 0x7f7fe013f710 2024-03-25T18:18:58.224 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.222+0000 7f7fe859a640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe00b28a0 con 0x7f7fe01430c0 2024-03-25T18:18:58.224 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.222+0000 7f7fe859a640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe00b2a30 con 0x7f7fe00a98f0 2024-03-25T18:18:58.224 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.222+0000 7f7fe859a640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:58.224 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.222+0000 7f7fe859a640 10 monclient(hunting): authenticate will time out at 7619.533293s 2024-03-25T18:18:58.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.239+0000 7f7fe6b37640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f7fe01430c0 0x7f7fe014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:60244/0 (socket says 172.21.15.156:60244) 2024-03-25T18:18:58.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.239+0000 7f7fe6b37640 1 -- 172.21.15.156:0/1634816383 learned_addr learned my addr 172.21.15.156:0/1634816383 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:18:58.254 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.253+0000 7f7fe5334640 1 -- 172.21.15.156:0/1634816383 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156642240 0 0) 0x7f7fe00b28a0 con 0x7f7fe01430c0 2024-03-25T18:18:58.254 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.253+0000 7f7fe5334640 10 monclient(hunting): _negotiate 2024-03-25T18:18:58.255 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.253+0000 7f7fe5334640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:58.255 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.253+0000 7f7fe5334640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:58.255 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.253+0000 7f7fe5334640 10 monclient(hunting): my global_id is 4708 2024-03-25T18:18:58.255 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.253+0000 7f7fe5334640 1 -- 172.21.15.156:0/1634816383 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fcc003520 con 0x7f7fe01430c0 2024-03-25T18:18:58.264 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.263+0000 7f7fe5334640 1 -- 172.21.15.156:0/1634816383 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3160160305 0 0) 0x7f7fe00b2a30 con 0x7f7fe00a98f0 2024-03-25T18:18:58.265 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.263+0000 7f7fe5334640 10 monclient(hunting): _negotiate 2024-03-25T18:18:58.265 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.263+0000 7f7fe5334640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:58.265 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.263+0000 7f7fe5334640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:58.265 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.263+0000 7f7fe5334640 10 monclient(hunting): my global_id is 4712 2024-03-25T18:18:58.265 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.263+0000 7f7fe5334640 1 -- 172.21.15.156:0/1634816383 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe00b28a0 con 0x7f7fe00a98f0 2024-03-25T18:18:58.266 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.264+0000 7f7fe5334640 1 -- 172.21.15.156:0/1634816383 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435891284 0 0) 0x7f7fe0142520 con 0x7f7fe013f710 2024-03-25T18:18:58.266 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.264+0000 7f7fe5334640 10 monclient(hunting): _negotiate 2024-03-25T18:18:58.266 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.264+0000 7f7fe5334640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:58.266 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.264+0000 7f7fe5334640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:58.266 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.264+0000 7f7fe5334640 10 monclient(hunting): my global_id is 4701 2024-03-25T18:18:58.266 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.264+0000 7f7fe5334640 1 -- 172.21.15.156:0/1634816383 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe00b2a30 con 0x7f7fe013f710 2024-03-25T18:18:58.271 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.270+0000 7f7fe5334640 1 -- 172.21.15.156:0/1634816383 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3223828334 0 0) 0x7f7fcc003520 con 0x7f7fe01430c0 2024-03-25T18:18:58.272 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.270+0000 7f7fe5334640 1 -- 172.21.15.156:0/1634816383 >> v1:172.21.15.156:6790/0 conn(0x7f7fe00a98f0 legacy=0x7f7fe00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:58.272 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.270+0000 7f7fe5334640 1 -- 172.21.15.156:0/1634816383 >> v1:172.21.15.156:6789/0 conn(0x7f7fe013f710 legacy=0x7f7fe01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:58.272 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.270+0000 7f7fe5334640 10 monclient: _finish_hunting 0 2024-03-25T18:18:58.272 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.270+0000 7f7fe5334640 1 monclient: found mon.noname-c 2024-03-25T18:18:58.272 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.270+0000 7f7fe5334640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:58.272 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.270+0000 7f7fe5334640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:18:58.272 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.270+0000 7f7fe5334640 1 -- 172.21.15.156:0/1634816383 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe00b44c0 con 0x7f7fe01430c0 2024-03-25T18:18:58.272 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.270+0000 7f7fe5334640 10 monclient: _finish_auth 0 2024-03-25T18:18:58.272 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.270+0000 7f7fe5334640 10 monclient: _check_auth_tickets 2024-03-25T18:18:58.272 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.270+0000 7f7fe5334640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:58.272 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.270+0000 7f7fe5334640 1 -- 172.21.15.156:0/1634816383 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7fd8001a10 con 0x7f7fe01430c0 2024-03-25T18:18:58.272 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.270+0000 7f7fe5334640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:58.272 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.270+0000 7f7fe5334640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:58.273 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.270+0000 7f7fe5334640 10 monclient: dump: 2024-03-25T18:18:58.273 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:58.273 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:58.273 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:58.273 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:58.273 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:58.273 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:58.273 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:58.273 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:58.273 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:58.273 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:58.273 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.270+0000 7f7fe859a640 5 monclient: authenticate success, global_id 4708 2024-03-25T18:18:58.273 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.270+0000 7f7fe859a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:58.276 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.275+0000 7f7fe5334640 1 -- 172.21.15.156:0/1634816383 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fd8001b70 con 0x7f7fe01430c0 2024-03-25T18:18:58.276 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.275+0000 7f7fe5334640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:58.276 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.275+0000 7f7fe5334640 1 -- 172.21.15.156:0/1634816383 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7fd8001f80 con 0x7f7fe01430c0 2024-03-25T18:18:58.276 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.275+0000 7f7fe859a640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:58.276 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.275+0000 7f7fe859a640 4 set_mon_vals no callback set 2024-03-25T18:18:58.276 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.275+0000 7f7fe5334640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:58.276 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.275+0000 7f7fe5334640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:58.277 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.275+0000 7f7fe5334640 10 monclient: dump: 2024-03-25T18:18:58.277 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:58.277 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:58.277 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:58.277 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:58.277 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:58.277 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:58.277 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:58.277 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:58.277 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:58.277 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:58.280 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.276+0000 7f7fe859a640 10 monclient: shutdown 2024-03-25T18:18:58.280 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.276+0000 7f7fe859a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:58.280 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.276+0000 7f7fe859a640 1 -- 172.21.15.156:0/1634816383 >> v1:172.21.15.194:6789/0 conn(0x7f7fe01430c0 legacy=0x7f7fe014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:58.280 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.276+0000 7f7fe859a640 1 -- 172.21.15.156:0/1634816383 shutdown_connections 2024-03-25T18:18:58.280 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.276+0000 7f7fe859a640 1 -- 172.21.15.156:0/1634816383 >> 172.21.15.156:0/1634816383 conn(0x7f7fe00b7ea0 msgr2=0x7f7fe00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:58.280 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.276+0000 7f7fe859a640 1 -- 172.21.15.156:0/1634816383 shutdown_connections 2024-03-25T18:18:58.280 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.276+0000 7f7fe859a640 1 -- 172.21.15.156:0/1634816383 wait complete. 2024-03-25T18:18:58.280 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.277+0000 7f7fe859a640 10 monclient: build_initial_monmap 2024-03-25T18:18:58.280 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.277+0000 7f7fe859a640 10 monclient: monmap: 2024-03-25T18:18:58.280 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:18:58.280 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:58.280 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:18:58.281 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:18:58.281 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:58.281 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:18:58.281 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:18:58.281 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:18:58.281 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:18:58.281 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:58.281 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.277+0000 7f7fe859a640 1 librados: starting msgr at 2024-03-25T18:18:58.281 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.277+0000 7f7fe859a640 1 librados: starting objecter 2024-03-25T18:18:58.281 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.277+0000 7f7fe859a640 1 Processor -- start 2024-03-25T18:18:58.281 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.277+0000 7f7fe859a640 1 -- start start 2024-03-25T18:18:58.281 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.277+0000 7f7fe859a640 1 librados: setting wanted keys 2024-03-25T18:18:58.281 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.277+0000 7f7fe859a640 1 librados: calling monclient init 2024-03-25T18:18:58.281 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.277+0000 7f7fe859a640 10 monclient: init 2024-03-25T18:18:58.281 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.278+0000 7f7fe859a640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:58.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.278+0000 7f7fe859a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:58.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.278+0000 7f7fe859a640 10 monclient(hunting): picked mon.noname-a con 0x7f7fe00a98f0 addr v1:172.21.15.156:6789/0 2024-03-25T18:18:58.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.278+0000 7f7fe859a640 10 monclient(hunting): picked mon.noname-c con 0x7f7fe013f710 addr v1:172.21.15.156:6790/0 2024-03-25T18:18:58.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.278+0000 7f7fe859a640 10 monclient(hunting): picked mon.noname-b con 0x7f7fe01430c0 addr v1:172.21.15.194:6789/0 2024-03-25T18:18:58.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.278+0000 7f7fe859a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe0135230 con 0x7f7fe00a98f0 2024-03-25T18:18:58.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.278+0000 7f7fe859a640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe0135da0 con 0x7f7fe01430c0 2024-03-25T18:18:58.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.278+0000 7f7fe859a640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe0135f30 con 0x7f7fe013f710 2024-03-25T18:18:58.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.278+0000 7f7fe859a640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:58.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.278+0000 7f7fe859a640 10 monclient(hunting): authenticate will time out at 7619.589146s 2024-03-25T18:18:58.288 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.287+0000 7f7fe5b35640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f7fe013f710 0x7f7fe01dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:52234/0 (socket says 172.21.15.156:52234) 2024-03-25T18:18:58.288 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.287+0000 7f7fe5b35640 1 -- 172.21.15.156:0/4255303898 learned_addr learned my addr 172.21.15.156:0/4255303898 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:18:58.290 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.289+0000 7f7fd6ffd640 10 client.?.objecter ms_handle_connect 0x7f7fe01430c0 2024-03-25T18:18:58.290 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.289+0000 7f7fd6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:58.294 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.293+0000 7f7fd6ffd640 10 client.?.objecter ms_handle_connect 0x7f7fe013f710 2024-03-25T18:18:58.294 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.293+0000 7f7fd6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:58.304 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.303+0000 7f7fd6ffd640 1 -- 172.21.15.156:0/4255303898 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851098815 0 0) 0x7f7fe0135da0 con 0x7f7fe01430c0 2024-03-25T18:18:58.304 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.303+0000 7f7fd6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:58.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.303+0000 7f7fd6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:58.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.303+0000 7f7fd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:58.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.303+0000 7f7fd6ffd640 10 monclient(hunting): my global_id is 4711 2024-03-25T18:18:58.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.303+0000 7f7fd6ffd640 1 -- 172.21.15.156:0/4255303898 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc4004eb0 con 0x7f7fe01430c0 2024-03-25T18:18:58.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.303+0000 7f7fd6ffd640 1 -- 172.21.15.156:0/4255303898 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2516539481 0 0) 0x7f7fe0135f30 con 0x7f7fe013f710 2024-03-25T18:18:58.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.303+0000 7f7fd6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:58.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.303+0000 7f7fd6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:58.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.303+0000 7f7fd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:58.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.303+0000 7f7fd6ffd640 10 monclient(hunting): my global_id is 4715 2024-03-25T18:18:58.305 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.303+0000 7f7fd6ffd640 1 -- 172.21.15.156:0/4255303898 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe0135da0 con 0x7f7fe013f710 2024-03-25T18:18:58.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.326+0000 7f7fd6ffd640 10 client.?.objecter ms_handle_connect 0x7f7fe00a98f0 2024-03-25T18:18:58.327 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.326+0000 7f7fd6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:58.388 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.387+0000 7f7fd6ffd640 1 -- 172.21.15.156:0/4255303898 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3672560018 0 0) 0x7f7fe0135230 con 0x7f7fe00a98f0 2024-03-25T18:18:58.389 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.387+0000 7f7fd6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:58.389 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.387+0000 7f7fd6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:58.389 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.387+0000 7f7fd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:58.389 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.387+0000 7f7fd6ffd640 10 monclient(hunting): my global_id is 4704 2024-03-25T18:18:58.389 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.387+0000 7f7fd6ffd640 1 -- 172.21.15.156:0/4255303898 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe0135f30 con 0x7f7fe00a98f0 2024-03-25T18:18:58.423 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.422+0000 7f7fd6ffd640 1 -- 172.21.15.156:0/4255303898 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2769268069 0 0) 0x7f7fe0135f30 con 0x7f7fe00a98f0 2024-03-25T18:18:58.423 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.422+0000 7f7fd6ffd640 1 -- 172.21.15.156:0/4255303898 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fe0135230 con 0x7f7fe00a98f0 2024-03-25T18:18:58.423 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.422+0000 7f7fd6ffd640 1 -- 172.21.15.156:0/4255303898 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7fdc001930 con 0x7f7fe00a98f0 2024-03-25T18:18:58.423 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.422+0000 7f7fd6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:58.423 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.422+0000 7f7fd6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:58.423 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.422+0000 7f7fd6ffd640 10 monclient(hunting): dump: 2024-03-25T18:18:58.423 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:58.423 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:58.424 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:58.424 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:58.424 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:58.424 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:58.424 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:58.424 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:58.424 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:58.424 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:58.424 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.422+0000 7f7fd6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:58.434 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.433+0000 7f7fd6ffd640 1 -- 172.21.15.156:0/4255303898 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4152385828 0 0) 0x7f7fc4004eb0 con 0x7f7fe01430c0 2024-03-25T18:18:58.434 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.433+0000 7f7fd6ffd640 1 -- 172.21.15.156:0/4255303898 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fe0135f30 con 0x7f7fe01430c0 2024-03-25T18:18:58.434 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.433+0000 7f7fd6ffd640 1 -- 172.21.15.156:0/4255303898 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7fd8001e80 con 0x7f7fe01430c0 2024-03-25T18:18:58.434 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.433+0000 7f7fd6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:58.434 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.433+0000 7f7fd6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:58.434 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.433+0000 7f7fd6ffd640 10 monclient(hunting): dump: 2024-03-25T18:18:58.434 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:58.434 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:58.434 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:58.435 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:58.435 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:58.435 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:58.435 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:58.435 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:58.435 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:58.435 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:58.442 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.441+0000 7f7fd6ffd640 1 -- 172.21.15.156:0/4255303898 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 651645817 0 0) 0x7f7fe0135da0 con 0x7f7fe013f710 2024-03-25T18:18:58.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.441+0000 7f7fd6ffd640 1 -- 172.21.15.156:0/4255303898 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fc4004eb0 con 0x7f7fe013f710 2024-03-25T18:18:58.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.441+0000 7f7fd6ffd640 1 -- 172.21.15.156:0/4255303898 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7fd00019f0 con 0x7f7fe013f710 2024-03-25T18:18:58.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.441+0000 7f7fd6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:58.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.441+0000 7f7fd6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:58.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.441+0000 7f7fd6ffd640 10 monclient(hunting): dump: 2024-03-25T18:18:58.443 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:58.443 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:58.443 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:58.443 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:58.443 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:58.443 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:58.443 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:58.443 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:58.443 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:58.443 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:58.451 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.449+0000 7f7fd6ffd640 1 -- 172.21.15.156:0/4255303898 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2131929869 0 0) 0x7f7fe0135230 con 0x7f7fe00a98f0 2024-03-25T18:18:58.451 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.449+0000 7f7fd6ffd640 1 -- 172.21.15.156:0/4255303898 >> v1:172.21.15.156:6790/0 conn(0x7f7fe013f710 legacy=0x7f7fe01dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:58.451 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.449+0000 7f7fd6ffd640 1 -- 172.21.15.156:0/4255303898 >> v1:172.21.15.194:6789/0 conn(0x7f7fe01430c0 legacy=0x7f7fe01db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:58.451 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.449+0000 7f7fd6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:58.451 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.449+0000 7f7fd6ffd640 1 monclient: found mon.a 2024-03-25T18:18:58.451 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.449+0000 7f7fd6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:58.451 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.449+0000 7f7fd6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:18:58.451 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.449+0000 7f7fd6ffd640 1 -- 172.21.15.156:0/4255303898 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe01360c0 con 0x7f7fe00a98f0 2024-03-25T18:18:58.451 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.449+0000 7f7fd6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:18:58.451 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.449+0000 7f7fd6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:58.451 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.449+0000 7f7fd6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:58.451 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.449+0000 7f7fe859a640 5 monclient: authenticate success, global_id 4704 2024-03-25T18:18:58.452 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.449+0000 7f7fe859a640 10 monclient: _renew_subs 2024-03-25T18:18:58.452 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.449+0000 7f7fe859a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:18:58.452 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.449+0000 7f7fe859a640 1 -- 172.21.15.156:0/4255303898 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe01d5000 con 0x7f7fe00a98f0 2024-03-25T18:18:58.452 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.449+0000 7f7fe859a640 10 client.4704.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:58.452 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.449+0000 7f7fe859a640 10 monclient: _renew_subs 2024-03-25T18:18:58.452 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.449+0000 7f7fe859a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:18:58.452 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.449+0000 7f7fe859a640 1 -- 172.21.15.156:0/4255303898 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe01d41e0 con 0x7f7fe00a98f0 2024-03-25T18:18:58.452 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.449+0000 7f7fe859a640 1 librados: init done 2024-03-25T18:18:58.453 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.450+0000 7f7fe859a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:58.453 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.451+0000 7f7fe859a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:58.453 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.451+0000 7f7fe859a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:18:58.453 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.451+0000 7f7fe859a640 1 -- 172.21.15.156:0/4255303898 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fe0135090 con 0x7f7fe00a98f0 2024-03-25T18:18:58.711 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.709+0000 7f7fd6ffd640 1 -- 172.21.15.156:0/4255303898 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fdc001d60 con 0x7f7fe00a98f0 2024-03-25T18:18:58.711 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.709+0000 7f7fd6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:58.711 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.709+0000 7f7fd6ffd640 1 -- 172.21.15.156:0/4255303898 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f7fdc0035f0 con 0x7f7fe00a98f0 2024-03-25T18:18:58.711 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.709+0000 7f7fd6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:58.711 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.709+0000 7f7fd6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:58.711 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.709+0000 7f7fd6ffd640 10 monclient: dump: 2024-03-25T18:18:58.711 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:18:58.711 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:18:58.711 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:58.711 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:18:58.711 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:58.712 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:18:58.712 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:18:58.712 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:18:58.712 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:18:58.712 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:18:58.712 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.709+0000 7f7fe4b33640 4 set_mon_vals no callback set 2024-03-25T18:18:58.712 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.710+0000 7f7fd6ffd640 1 -- 172.21.15.156:0/4255303898 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f7fdc019680 con 0x7f7fe00a98f0 2024-03-25T18:18:58.713 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.711+0000 7f7fe5b35640 10 monclient: get_auth_request con 0x7f7fc406e0f0 auth_method 0 2024-03-25T18:18:58.713 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.711+0000 7f7fd6ffd640 10 client.4704.objecter ms_handle_connect 0x7f7fc406e0f0 2024-03-25T18:18:58.716 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.715+0000 7f7fd6ffd640 1 -- 172.21.15.156:0/4255303898 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3381691338 0 0) 0x7f7fdc0532e0 con 0x7f7fe00a98f0 2024-03-25T18:18:58.717 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.715+0000 7f7fd6ffd640 10 client.4704.objecter ms_dispatch 0x7f7fe0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:18:58.717 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.715+0000 7f7fd6ffd640 3 client.4704.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:18:58.717 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.715+0000 7f7fd6ffd640 3 client.4704.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:18:58.717 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.715+0000 7f7fd6ffd640 20 client.4704.objecter dump_active .. 0 homeless 2024-03-25T18:18:58.717 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.715+0000 7f7fd6ffd640 1 -- 172.21.15.156:0/4255303898 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7fdc04ec20 con 0x7f7fe00a98f0 2024-03-25T18:18:58.717 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.715+0000 7f7fd6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:58.717 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.715+0000 7f7fd6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:58.962 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.961+0000 7f7fe859a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:58.963 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.961+0000 7f7fe859a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:58.963 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.961+0000 7f7fe859a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:18:58.963 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:58.961+0000 7f7fe859a640 1 -- 172.21.15.156:0/4255303898 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fe013b700 con 0x7f7fe00a98f0 2024-03-25T18:18:59.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:59.013+0000 7f7fd6ffd640 1 -- 172.21.15.156:0/4255303898 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1441834839) 0x7f7fdc050d20 con 0x7f7fe00a98f0 2024-03-25T18:18:59.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:59.013+0000 7f7fd6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:59.014 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:59.013+0000 7f7fd6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:59.015 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:18:59.015 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":11,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:18:20.029906+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:18:20.029906+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:59.017 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:59.015+0000 7f7fe859a640 10 librados: watch_flush enter 2024-03-25T18:18:59.018 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:59.015+0000 7f7fe859a640 10 librados: watch_flush exit 2024-03-25T18:18:59.018 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:59.015+0000 7f7fe859a640 20 client.4704.objecter shutdown clearing up homeless session... 2024-03-25T18:18:59.018 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:59.015+0000 7f7fe859a640 10 client.4704.objecter successfully canceled tick 2024-03-25T18:18:59.018 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:59.016+0000 7f7fe859a640 1 -- 172.21.15.156:0/4255303898 >> v1:172.21.15.194:6800/2946427713 conn(0x7f7fc406e0f0 legacy=0x7f7fc40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:59.018 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:59.016+0000 7f7fe859a640 10 monclient: shutdown 2024-03-25T18:18:59.018 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:59.016+0000 7f7fe859a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:59.018 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:59.016+0000 7f7fe859a640 1 -- 172.21.15.156:0/4255303898 >> v1:172.21.15.156:6789/0 conn(0x7f7fe00a98f0 legacy=0x7f7fe01d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:59.018 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:59.016+0000 7f7fe859a640 1 -- 172.21.15.156:0/4255303898 shutdown_connections 2024-03-25T18:18:59.018 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:59.016+0000 7f7fe859a640 1 -- 172.21.15.156:0/4255303898 >> 172.21.15.156:0/4255303898 conn(0x7f7fe00b7ea0 msgr2=0x7f7fe014ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:59.018 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:59.016+0000 7f7fe859a640 1 -- 172.21.15.156:0/4255303898 shutdown_connections 2024-03-25T18:18:59.019 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:59.016+0000 7f7fe859a640 1 -- 172.21.15.156:0/4255303898 wait complete. 2024-03-25T18:18:59.019 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:18:59.016+0000 7f7fe859a640 1 librados: shutdown 2024-03-25T18:18:59.030 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:00.282 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:11:06.272 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:00.288 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:11:06.278 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:00.320 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:11:06.310 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:00.339 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:11:06.306 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:00.349 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:11:06.339 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:00.357 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:11:06.347 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:00.359 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:11:06.326 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:00.384 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:11:06.351 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:00.387 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:11:06.377 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:05.836 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:05.950 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.949+0000 7fd16fd20640 10 monclient: get_monmap_and_config 2024-03-25T18:19:05.950 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.949+0000 7fd16fd20640 10 monclient: build_initial_monmap 2024-03-25T18:19:05.950 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.949+0000 7fd16fd20640 10 monclient: monmap: 2024-03-25T18:19:05.950 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:19:05.950 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:05.950 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:19:05.950325+0000 2024-03-25T18:19:05.950 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:19:05.950325+0000 2024-03-25T18:19:05.951 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:05.951 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:05.951 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:19:05.951 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:19:05.951 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:19:05.951 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:05.952 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.950+0000 7fd16fd20640 1 Processor -- start 2024-03-25T18:19:05.952 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.950+0000 7fd16fd20640 1 -- start start 2024-03-25T18:19:05.952 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.950+0000 7fd16fd20640 10 monclient: init 2024-03-25T18:19:05.952 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.950+0000 7fd16fd20640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:05.952 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.950+0000 7fd16fd20640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:05.952 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.950+0000 7fd16fd20640 10 monclient(hunting): picked mon.noname-c con 0x7fd1680705b0 addr v1:172.21.15.194:6789/0 2024-03-25T18:19:05.952 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.951+0000 7fd16fd20640 10 monclient(hunting): picked mon.noname-b con 0x7fd1681402d0 addr v1:172.21.15.156:6790/0 2024-03-25T18:19:05.954 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.951+0000 7fd16fd20640 10 monclient(hunting): picked mon.noname-a con 0x7fd168149ae0 addr v1:172.21.15.156:6789/0 2024-03-25T18:19:05.954 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.951+0000 7fd16fd20640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1680718c0 con 0x7fd168149ae0 2024-03-25T18:19:05.954 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.951+0000 7fd16fd20640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1680714c0 con 0x7fd1680705b0 2024-03-25T18:19:05.954 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.951+0000 7fd16fd20640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd168072b50 con 0x7fd1681402d0 2024-03-25T18:19:05.954 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.951+0000 7fd16fd20640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:05.954 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.951+0000 7fd16fd20640 10 monclient(hunting): authenticate will time out at 7627.261934s 2024-03-25T18:19:05.965 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.964+0000 7fd16e2bd640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fd168149ae0 0x7fd16814bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:35746/0 (socket says 172.21.15.156:35746) 2024-03-25T18:19:05.965 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.964+0000 7fd16e2bd640 1 -- 172.21.15.156:0/3906991030 learned_addr learned my addr 172.21.15.156:0/3906991030 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:19:05.979 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.978+0000 7fd16caba640 1 -- 172.21.15.156:0/3906991030 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694070230 0 0) 0x7fd168072b50 con 0x7fd1681402d0 2024-03-25T18:19:05.979 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.978+0000 7fd16caba640 10 monclient(hunting): _negotiate 2024-03-25T18:19:05.980 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.978+0000 7fd16caba640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:05.980 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.978+0000 7fd16caba640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:05.980 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.978+0000 7fd16caba640 10 monclient(hunting): my global_id is 4718 2024-03-25T18:19:05.980 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.978+0000 7fd16caba640 1 -- 172.21.15.156:0/3906991030 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd154003520 con 0x7fd1681402d0 2024-03-25T18:19:05.982 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.981+0000 7fd16caba640 1 -- 172.21.15.156:0/3906991030 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1728156943 0 0) 0x7fd1680718c0 con 0x7fd168149ae0 2024-03-25T18:19:05.982 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.981+0000 7fd16caba640 10 monclient(hunting): _negotiate 2024-03-25T18:19:05.982 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.981+0000 7fd16caba640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:05.982 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.981+0000 7fd16caba640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:05.982 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.981+0000 7fd16caba640 10 monclient(hunting): my global_id is 4707 2024-03-25T18:19:05.983 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.981+0000 7fd16caba640 1 -- 172.21.15.156:0/3906991030 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd168072b50 con 0x7fd168149ae0 2024-03-25T18:19:05.984 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.983+0000 7fd16caba640 1 -- 172.21.15.156:0/3906991030 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501682154 0 0) 0x7fd1680714c0 con 0x7fd1680705b0 2024-03-25T18:19:05.984 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.983+0000 7fd16caba640 10 monclient(hunting): _negotiate 2024-03-25T18:19:05.984 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.983+0000 7fd16caba640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:05.984 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.983+0000 7fd16caba640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:05.984 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.983+0000 7fd16caba640 10 monclient(hunting): my global_id is 4714 2024-03-25T18:19:05.984 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.983+0000 7fd16caba640 1 -- 172.21.15.156:0/3906991030 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1680718c0 con 0x7fd1680705b0 2024-03-25T18:19:05.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.993+0000 7fd16caba640 1 -- 172.21.15.156:0/3906991030 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 834855863 0 0) 0x7fd168072b50 con 0x7fd168149ae0 2024-03-25T18:19:05.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.993+0000 7fd16caba640 1 -- 172.21.15.156:0/3906991030 >> v1:172.21.15.156:6790/0 conn(0x7fd1681402d0 legacy=0x7fd168140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:05.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.993+0000 7fd16caba640 1 -- 172.21.15.156:0/3906991030 >> v1:172.21.15.194:6789/0 conn(0x7fd1680705b0 legacy=0x7fd16813d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:05.994 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.993+0000 7fd16caba640 10 monclient: _finish_hunting 0 2024-03-25T18:19:05.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.993+0000 7fd16caba640 1 monclient: found mon.noname-a 2024-03-25T18:19:05.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.993+0000 7fd16caba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:05.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.993+0000 7fd16caba640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:19:05.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.993+0000 7fd16caba640 1 -- 172.21.15.156:0/3906991030 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16813dd10 con 0x7fd168149ae0 2024-03-25T18:19:05.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.993+0000 7fd16caba640 10 monclient: _finish_auth 0 2024-03-25T18:19:05.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.993+0000 7fd16caba640 10 monclient: _check_auth_tickets 2024-03-25T18:19:05.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.993+0000 7fd16caba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:05.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.993+0000 7fd16caba640 1 -- 172.21.15.156:0/3906991030 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fd1600019f0 con 0x7fd168149ae0 2024-03-25T18:19:05.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.993+0000 7fd16fd20640 5 monclient: authenticate success, global_id 4707 2024-03-25T18:19:05.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.993+0000 7fd16fd20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:05.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.993+0000 7fd16caba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:05.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.993+0000 7fd16caba640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:05.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.993+0000 7fd16caba640 10 monclient: dump: 2024-03-25T18:19:05.995 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:19:05.995 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:05.996 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:05.996 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:05.996 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:05.996 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:05.996 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:19:05.996 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:19:05.996 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:19:05.996 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:05.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.993+0000 7fd16fd20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:05.999 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.998+0000 7fd16caba640 1 -- 172.21.15.156:0/3906991030 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd160001b50 con 0x7fd168149ae0 2024-03-25T18:19:05.999 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.998+0000 7fd16caba640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:05.999 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.998+0000 7fd16caba640 1 -- 172.21.15.156:0/3906991030 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fd160001f60 con 0x7fd168149ae0 2024-03-25T18:19:05.999 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.998+0000 7fd16fd20640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:05.999 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.998+0000 7fd16fd20640 4 set_mon_vals no callback set 2024-03-25T18:19:05.999 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.998+0000 7fd16caba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:05.999 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.998+0000 7fd16caba640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:06.000 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.998+0000 7fd16caba640 10 monclient: dump: 2024-03-25T18:19:06.000 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:19:06.000 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:06.000 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:06.000 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:06.000 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:06.000 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:06.000 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:19:06.000 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:19:06.000 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:19:06.000 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:06.002 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.998+0000 7fd16fd20640 10 monclient: shutdown 2024-03-25T18:19:06.002 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.998+0000 7fd16fd20640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:06.002 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.998+0000 7fd16fd20640 1 -- 172.21.15.156:0/3906991030 >> v1:172.21.15.156:6789/0 conn(0x7fd168149ae0 legacy=0x7fd16814bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:06.002 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.998+0000 7fd16fd20640 1 -- 172.21.15.156:0/3906991030 shutdown_connections 2024-03-25T18:19:06.002 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.998+0000 7fd16fd20640 1 -- 172.21.15.156:0/3906991030 >> 172.21.15.156:0/3906991030 conn(0x7fd1680b5d10 msgr2=0x7fd1680b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:06.003 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.999+0000 7fd16fd20640 1 -- 172.21.15.156:0/3906991030 shutdown_connections 2024-03-25T18:19:06.003 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.999+0000 7fd16fd20640 1 -- 172.21.15.156:0/3906991030 wait complete. 2024-03-25T18:19:06.003 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.999+0000 7fd16fd20640 10 monclient: build_initial_monmap 2024-03-25T18:19:06.003 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:05.999+0000 7fd16fd20640 10 monclient: monmap: 2024-03-25T18:19:06.003 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:19:06.003 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:06.003 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:19:06.003 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:19:06.003 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:06.003 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:19:06.003 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:19:06.003 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:19:06.004 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:19:06.004 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:06.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.000+0000 7fd16fd20640 1 librados: starting msgr at 2024-03-25T18:19:06.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.000+0000 7fd16fd20640 1 librados: starting objecter 2024-03-25T18:19:06.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.000+0000 7fd16fd20640 1 Processor -- start 2024-03-25T18:19:06.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.000+0000 7fd16fd20640 1 -- start start 2024-03-25T18:19:06.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.000+0000 7fd16fd20640 1 librados: setting wanted keys 2024-03-25T18:19:06.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.000+0000 7fd16fd20640 1 librados: calling monclient init 2024-03-25T18:19:06.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.000+0000 7fd16fd20640 10 monclient: init 2024-03-25T18:19:06.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.000+0000 7fd16fd20640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:06.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.000+0000 7fd16fd20640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:06.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.000+0000 7fd16fd20640 10 monclient(hunting): picked mon.noname-a con 0x7fd1680705b0 addr v1:172.21.15.156:6789/0 2024-03-25T18:19:06.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.000+0000 7fd16fd20640 10 monclient(hunting): picked mon.noname-c con 0x7fd1681402d0 addr v1:172.21.15.156:6790/0 2024-03-25T18:19:06.004 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.001+0000 7fd16fd20640 10 monclient(hunting): picked mon.noname-b con 0x7fd168149ae0 addr v1:172.21.15.194:6789/0 2024-03-25T18:19:06.005 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.001+0000 7fd16fd20640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd168141cd0 con 0x7fd1680705b0 2024-03-25T18:19:06.005 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.001+0000 7fd16fd20640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16813e9e0 con 0x7fd168149ae0 2024-03-25T18:19:06.005 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.001+0000 7fd16fd20640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16813f610 con 0x7fd1681402d0 2024-03-25T18:19:06.005 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.001+0000 7fd16fd20640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:06.005 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.001+0000 7fd16fd20640 10 monclient(hunting): authenticate will time out at 7627.311919s 2024-03-25T18:19:06.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.008+0000 7fd16dabc640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fd1680705b0 0x7fd16813ef00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:35790/0 (socket says 172.21.15.156:35790) 2024-03-25T18:19:06.009 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.008+0000 7fd16dabc640 1 -- 172.21.15.156:0/3751691467 learned_addr learned my addr 172.21.15.156:0/3751691467 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:19:06.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.009+0000 7fd15e7fc640 10 client.?.objecter ms_handle_connect 0x7fd1680705b0 2024-03-25T18:19:06.010 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.009+0000 7fd15e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:06.018 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.017+0000 7fd15e7fc640 10 client.?.objecter ms_handle_connect 0x7fd168149ae0 2024-03-25T18:19:06.018 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.017+0000 7fd15e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:06.019 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.018+0000 7fd15e7fc640 1 -- 172.21.15.156:0/3751691467 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1481324585 0 0) 0x7fd168141cd0 con 0x7fd1680705b0 2024-03-25T18:19:06.019 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.018+0000 7fd15e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:06.019 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.018+0000 7fd15e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:06.019 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.018+0000 7fd15e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:06.019 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.018+0000 7fd15e7fc640 10 monclient(hunting): my global_id is 4710 2024-03-25T18:19:06.019 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.018+0000 7fd15e7fc640 1 -- 172.21.15.156:0/3751691467 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd14c004eb0 con 0x7fd1680705b0 2024-03-25T18:19:06.021 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.020+0000 7fd15e7fc640 10 client.?.objecter ms_handle_connect 0x7fd1681402d0 2024-03-25T18:19:06.021 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.020+0000 7fd15e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:06.025 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.024+0000 7fd15e7fc640 1 -- 172.21.15.156:0/3751691467 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59387702 0 0) 0x7fd16813e9e0 con 0x7fd168149ae0 2024-03-25T18:19:06.025 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.024+0000 7fd15e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:06.025 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.024+0000 7fd15e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:06.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.024+0000 7fd15e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:06.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.024+0000 7fd15e7fc640 10 monclient(hunting): my global_id is 4717 2024-03-25T18:19:06.026 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.024+0000 7fd15e7fc640 1 -- 172.21.15.156:0/3751691467 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd168141cd0 con 0x7fd168149ae0 2024-03-25T18:19:06.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.026+0000 7fd15e7fc640 1 -- 172.21.15.156:0/3751691467 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1542613459 0 0) 0x7fd16813f610 con 0x7fd1681402d0 2024-03-25T18:19:06.027 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.026+0000 7fd15e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:06.028 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.026+0000 7fd15e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:06.028 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.026+0000 7fd15e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:06.028 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.026+0000 7fd15e7fc640 10 monclient(hunting): my global_id is 4721 2024-03-25T18:19:06.028 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.026+0000 7fd15e7fc640 1 -- 172.21.15.156:0/3751691467 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd16813e9e0 con 0x7fd1681402d0 2024-03-25T18:19:06.030 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.029+0000 7fd15e7fc640 1 -- 172.21.15.156:0/3751691467 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 592926684 0 0) 0x7fd14c004eb0 con 0x7fd1680705b0 2024-03-25T18:19:06.031 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.029+0000 7fd15e7fc640 1 -- 172.21.15.156:0/3751691467 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd16813f610 con 0x7fd1680705b0 2024-03-25T18:19:06.031 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.029+0000 7fd15e7fc640 1 -- 172.21.15.156:0/3751691467 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fd164001a10 con 0x7fd1680705b0 2024-03-25T18:19:06.031 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.029+0000 7fd15e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:06.031 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.029+0000 7fd15e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:06.031 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.029+0000 7fd15e7fc640 10 monclient(hunting): dump: 2024-03-25T18:19:06.031 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:19:06.031 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:06.031 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:06.031 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:06.031 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:06.031 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:06.031 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:19:06.031 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:19:06.031 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:19:06.031 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:06.031 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.029+0000 7fd15e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:06.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.040+0000 7fd15e7fc640 1 -- 172.21.15.156:0/3751691467 <== mon.0 v1:172.21.15.156:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3945322433 0 0) 0x7fd16813f610 con 0x7fd1680705b0 2024-03-25T18:19:06.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.040+0000 7fd15e7fc640 1 -- 172.21.15.156:0/3751691467 >> v1:172.21.15.156:6790/0 conn(0x7fd1681402d0 legacy=0x7fd1681d8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:06.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.040+0000 7fd15e7fc640 1 -- 172.21.15.156:0/3751691467 >> v1:172.21.15.194:6789/0 conn(0x7fd168149ae0 legacy=0x7fd1681d9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:06.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.040+0000 7fd15e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:06.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.040+0000 7fd15e7fc640 1 monclient: found mon.a 2024-03-25T18:19:06.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.040+0000 7fd15e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:06.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.040+0000 7fd15e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:19:06.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.040+0000 7fd15e7fc640 1 -- 172.21.15.156:0/3751691467 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16813f7a0 con 0x7fd1680705b0 2024-03-25T18:19:06.042 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.040+0000 7fd15e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:06.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.040+0000 7fd15e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:06.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.040+0000 7fd15e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:06.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.040+0000 7fd16fd20640 5 monclient: authenticate success, global_id 4710 2024-03-25T18:19:06.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.040+0000 7fd16fd20640 10 monclient: _renew_subs 2024-03-25T18:19:06.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.040+0000 7fd16fd20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:19:06.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.040+0000 7fd16fd20640 1 -- 172.21.15.156:0/3751691467 --> v1:172.21.15.156:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd168136030 con 0x7fd1680705b0 2024-03-25T18:19:06.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.041+0000 7fd16fd20640 10 client.4710.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:06.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.041+0000 7fd16fd20640 10 monclient: _renew_subs 2024-03-25T18:19:06.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.041+0000 7fd16fd20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:19:06.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.041+0000 7fd16fd20640 1 -- 172.21.15.156:0/3751691467 --> v1:172.21.15.156:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1681e1120 con 0x7fd1680705b0 2024-03-25T18:19:06.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.041+0000 7fd16fd20640 1 librados: init done 2024-03-25T18:19:06.044 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.042+0000 7fd16fd20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:06.044 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.042+0000 7fd16fd20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:06.044 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.042+0000 7fd16fd20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:19:06.044 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.042+0000 7fd16fd20640 1 -- 172.21.15.156:0/3751691467 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1680627b0 con 0x7fd1680705b0 2024-03-25T18:19:06.382 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.381+0000 7fd15e7fc640 1 -- 172.21.15.156:0/3751691467 <== mon.0 v1:172.21.15.156:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd164001c70 con 0x7fd1680705b0 2024-03-25T18:19:06.382 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.381+0000 7fd15e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:06.382 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.381+0000 7fd15e7fc640 1 -- 172.21.15.156:0/3751691467 <== mon.0 v1:172.21.15.156:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7fd1640038c0 con 0x7fd1680705b0 2024-03-25T18:19:06.383 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.381+0000 7fd15e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:06.383 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.381+0000 7fd15e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:06.383 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.381+0000 7fd15e7fc640 10 monclient: dump: 2024-03-25T18:19:06.383 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:19:06.383 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:06.383 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:06.383 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:06.383 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:06.383 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:06.383 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:19:06.383 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:19:06.383 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:19:06.383 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:06.383 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.381+0000 7fd15ffff640 4 set_mon_vals no callback set 2024-03-25T18:19:06.384 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.382+0000 7fd15e7fc640 1 -- 172.21.15.156:0/3751691467 <== mon.0 v1:172.21.15.156:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7fd164003ae0 con 0x7fd1680705b0 2024-03-25T18:19:06.384 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.382+0000 7fd16d2bb640 10 monclient: get_auth_request con 0x7fd14c06ddf0 auth_method 0 2024-03-25T18:19:06.384 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.383+0000 7fd15e7fc640 10 client.4710.objecter ms_handle_connect 0x7fd14c06ddf0 2024-03-25T18:19:06.390 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.389+0000 7fd15e7fc640 1 -- 172.21.15.156:0/3751691467 <== mon.0 v1:172.21.15.156:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3381691338 0 0) 0x7fd164054710 con 0x7fd1680705b0 2024-03-25T18:19:06.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.389+0000 7fd15e7fc640 10 client.4710.objecter ms_dispatch 0x7fd168000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:19:06.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.389+0000 7fd15e7fc640 3 client.4710.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:19:06.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.389+0000 7fd15e7fc640 3 client.4710.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:19:06.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.389+0000 7fd15e7fc640 20 client.4710.objecter dump_active .. 0 homeless 2024-03-25T18:19:06.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.389+0000 7fd15e7fc640 1 -- 172.21.15.156:0/3751691467 <== mon.0 v1:172.21.15.156:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd164050220 con 0x7fd1680705b0 2024-03-25T18:19:06.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.389+0000 7fd15e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:06.391 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.389+0000 7fd15e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:06.614 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.612+0000 7fd16fd20640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:06.614 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.612+0000 7fd16fd20640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:06.614 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.612+0000 7fd16fd20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.156:6789/0 2024-03-25T18:19:06.614 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.612+0000 7fd16fd20640 1 -- 172.21.15.156:0/3751691467 --> v1:172.21.15.156:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd16813eb90 con 0x7fd1680705b0 2024-03-25T18:19:06.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.699+0000 7fd15e7fc640 1 -- 172.21.15.156:0/3751691467 <== mon.0 v1:172.21.15.156:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1441834839) 0x7fd16813eb90 con 0x7fd1680705b0 2024-03-25T18:19:06.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.699+0000 7fd15e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:06.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.699+0000 7fd15e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:06.700 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:19:06.701 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":11,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:18:20.029906+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:18:20.029906+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:06.703 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.701+0000 7fd16fd20640 10 librados: watch_flush enter 2024-03-25T18:19:06.703 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.701+0000 7fd16fd20640 10 librados: watch_flush exit 2024-03-25T18:19:06.703 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.701+0000 7fd16fd20640 20 client.4710.objecter shutdown clearing up homeless session... 2024-03-25T18:19:06.703 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.701+0000 7fd16fd20640 10 client.4710.objecter successfully canceled tick 2024-03-25T18:19:06.703 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.702+0000 7fd16fd20640 1 -- 172.21.15.156:0/3751691467 >> v1:172.21.15.194:6800/2946427713 conn(0x7fd14c06ddf0 legacy=0x7fd14c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:06.703 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.702+0000 7fd16fd20640 10 monclient: shutdown 2024-03-25T18:19:06.703 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.702+0000 7fd16fd20640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:06.703 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.702+0000 7fd16fd20640 1 -- 172.21.15.156:0/3751691467 >> v1:172.21.15.156:6789/0 conn(0x7fd1680705b0 legacy=0x7fd16813ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:06.704 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.702+0000 7fd16fd20640 1 -- 172.21.15.156:0/3751691467 shutdown_connections 2024-03-25T18:19:06.704 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.702+0000 7fd16fd20640 1 -- 172.21.15.156:0/3751691467 >> 172.21.15.156:0/3751691467 conn(0x7fd1680b5d10 msgr2=0x7fd1680b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:06.704 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.702+0000 7fd16fd20640 1 -- 172.21.15.156:0/3751691467 shutdown_connections 2024-03-25T18:19:06.704 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.702+0000 7fd16fd20640 1 -- 172.21.15.156:0/3751691467 wait complete. 2024-03-25T18:19:06.704 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:06.702+0000 7fd16fd20640 1 librados: shutdown 2024-03-25T18:19:06.714 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:13.459 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:11:19.449 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:13.465 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:11:19.455 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:13.473 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:11:19.440 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:13.476 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:11:19.443 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:13.485 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:11:19.475 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:13.485 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:11:19.475 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:13.488 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:11:19.479 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:13.494 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:11:19.461 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:13.508 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:11:19.498 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:13.521 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:13.628 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.627+0000 7f5f468af640 10 monclient: get_monmap_and_config 2024-03-25T18:19:13.628 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.627+0000 7f5f468af640 10 monclient: build_initial_monmap 2024-03-25T18:19:13.628 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.627+0000 7f5f468af640 10 monclient: monmap: 2024-03-25T18:19:13.629 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:19:13.629 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:13.629 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:19:13.628705+0000 2024-03-25T18:19:13.629 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:19:13.628705+0000 2024-03-25T18:19:13.629 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:13.629 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:13.629 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:19:13.629 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:19:13.629 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:19:13.629 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:13.630 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.628+0000 7f5f468af640 1 Processor -- start 2024-03-25T18:19:13.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.628+0000 7f5f468af640 1 -- start start 2024-03-25T18:19:13.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.628+0000 7f5f468af640 10 monclient: init 2024-03-25T18:19:13.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.629+0000 7f5f468af640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:13.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.629+0000 7f5f468af640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:13.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.629+0000 7f5f468af640 10 monclient(hunting): picked mon.noname-c con 0x7f5f400a98f0 addr v1:172.21.15.194:6789/0 2024-03-25T18:19:13.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.629+0000 7f5f468af640 10 monclient(hunting): picked mon.noname-a con 0x7f5f4013f710 addr v1:172.21.15.156:6789/0 2024-03-25T18:19:13.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.629+0000 7f5f468af640 10 monclient(hunting): picked mon.noname-b con 0x7f5f401430c0 addr v1:172.21.15.156:6790/0 2024-03-25T18:19:13.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.629+0000 7f5f468af640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f40142520 con 0x7f5f4013f710 2024-03-25T18:19:13.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.629+0000 7f5f468af640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f400b28a0 con 0x7f5f400a98f0 2024-03-25T18:19:13.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.629+0000 7f5f468af640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f400b2a30 con 0x7f5f401430c0 2024-03-25T18:19:13.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.629+0000 7f5f468af640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:13.631 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.629+0000 7f5f468af640 10 monclient(hunting): authenticate will time out at 7634.940119s 2024-03-25T18:19:13.643 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.642+0000 7f5f44e4c640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f5f401430c0 0x7f5f4014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:55198/0 (socket says 172.21.15.156:55198) 2024-03-25T18:19:13.643 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.642+0000 7f5f44e4c640 1 -- 172.21.15.156:0/692780012 learned_addr learned my addr 172.21.15.156:0/692780012 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:19:13.673 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.672+0000 7f5f3effd640 1 -- 172.21.15.156:0/692780012 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722701564 0 0) 0x7f5f400b2a30 con 0x7f5f401430c0 2024-03-25T18:19:13.673 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.672+0000 7f5f3effd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:13.673 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.672+0000 7f5f3effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:13.673 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.672+0000 7f5f3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:13.673 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.672+0000 7f5f3effd640 10 monclient(hunting): my global_id is 4724 2024-03-25T18:19:13.673 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.672+0000 7f5f3effd640 1 -- 172.21.15.156:0/692780012 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f28003520 con 0x7f5f401430c0 2024-03-25T18:19:13.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.699+0000 7f5f3effd640 1 -- 172.21.15.156:0/692780012 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1245513550 0 0) 0x7f5f28003520 con 0x7f5f401430c0 2024-03-25T18:19:13.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.699+0000 7f5f3effd640 1 -- 172.21.15.156:0/692780012 >> v1:172.21.15.194:6789/0 conn(0x7f5f400a98f0 legacy=0x7f5f400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:13.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.699+0000 7f5f3effd640 1 -- 172.21.15.156:0/692780012 >> v1:172.21.15.156:6789/0 conn(0x7f5f4013f710 legacy=0x7f5f401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:13.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.699+0000 7f5f3effd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:13.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.699+0000 7f5f3effd640 1 monclient: found mon.noname-b 2024-03-25T18:19:13.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.699+0000 7f5f3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:13.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.699+0000 7f5f3effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.156:6790/0 2024-03-25T18:19:13.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.699+0000 7f5f3effd640 1 -- 172.21.15.156:0/692780012 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f400b44c0 con 0x7f5f401430c0 2024-03-25T18:19:13.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.699+0000 7f5f3effd640 10 monclient: _finish_auth 0 2024-03-25T18:19:13.700 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.699+0000 7f5f3effd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:13.701 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.699+0000 7f5f3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:13.701 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.699+0000 7f5f3effd640 1 -- 172.21.15.156:0/692780012 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5f38001a10 con 0x7f5f401430c0 2024-03-25T18:19:13.701 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.699+0000 7f5f468af640 5 monclient: authenticate success, global_id 4724 2024-03-25T18:19:13.701 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.699+0000 7f5f468af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:13.701 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.699+0000 7f5f3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:13.701 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.699+0000 7f5f3effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:13.701 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.699+0000 7f5f3effd640 10 monclient: dump: 2024-03-25T18:19:13.701 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:19:13.701 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:13.701 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:13.701 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:13.701 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:13.701 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:13.701 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:19:13.701 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:19:13.702 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:19:13.702 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:13.702 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.699+0000 7f5f468af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:13.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.739+0000 7f5f3effd640 1 -- 172.21.15.156:0/692780012 <== mon.2 v1:172.21.15.156:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f38001b70 con 0x7f5f401430c0 2024-03-25T18:19:13.740 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.739+0000 7f5f3effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:13.741 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.739+0000 7f5f3effd640 1 -- 172.21.15.156:0/692780012 <== mon.2 v1:172.21.15.156:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5f38001f80 con 0x7f5f401430c0 2024-03-25T18:19:13.741 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.739+0000 7f5f468af640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:13.741 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.739+0000 7f5f468af640 4 set_mon_vals no callback set 2024-03-25T18:19:13.741 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.739+0000 7f5f3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:13.741 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.739+0000 7f5f3effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:13.741 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.739+0000 7f5f3effd640 10 monclient: dump: 2024-03-25T18:19:13.741 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:19:13.741 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:13.741 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:13.741 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:13.741 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:13.741 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:13.741 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:19:13.741 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:19:13.741 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:19:13.741 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:13.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.740+0000 7f5f468af640 10 monclient: shutdown 2024-03-25T18:19:13.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.740+0000 7f5f468af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:13.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.740+0000 7f5f468af640 1 -- 172.21.15.156:0/692780012 >> v1:172.21.15.156:6790/0 conn(0x7f5f401430c0 legacy=0x7f5f4014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:13.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.740+0000 7f5f468af640 1 -- 172.21.15.156:0/692780012 shutdown_connections 2024-03-25T18:19:13.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.740+0000 7f5f468af640 1 -- 172.21.15.156:0/692780012 >> 172.21.15.156:0/692780012 conn(0x7f5f400b7ea0 msgr2=0x7f5f400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:13.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.740+0000 7f5f468af640 1 -- 172.21.15.156:0/692780012 shutdown_connections 2024-03-25T18:19:13.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.740+0000 7f5f468af640 1 -- 172.21.15.156:0/692780012 wait complete. 2024-03-25T18:19:13.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.741+0000 7f5f468af640 10 monclient: build_initial_monmap 2024-03-25T18:19:13.743 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.741+0000 7f5f468af640 10 monclient: monmap: 2024-03-25T18:19:13.743 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:19:13.744 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:13.744 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:19:13.744 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:19:13.744 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:13.744 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:19:13.744 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:19:13.744 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:19:13.744 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:19:13.744 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:13.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.741+0000 7f5f468af640 1 librados: starting msgr at 2024-03-25T18:19:13.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.741+0000 7f5f468af640 1 librados: starting objecter 2024-03-25T18:19:13.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.742+0000 7f5f468af640 1 Processor -- start 2024-03-25T18:19:13.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.742+0000 7f5f468af640 1 -- start start 2024-03-25T18:19:13.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.742+0000 7f5f468af640 1 librados: setting wanted keys 2024-03-25T18:19:13.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.742+0000 7f5f468af640 1 librados: calling monclient init 2024-03-25T18:19:13.744 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.742+0000 7f5f468af640 10 monclient: init 2024-03-25T18:19:13.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.742+0000 7f5f468af640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:13.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.742+0000 7f5f468af640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:13.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.742+0000 7f5f468af640 10 monclient(hunting): picked mon.noname-c con 0x7f5f400a98f0 addr v1:172.21.15.156:6790/0 2024-03-25T18:19:13.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.743+0000 7f5f468af640 10 monclient(hunting): picked mon.noname-b con 0x7f5f4013f710 addr v1:172.21.15.194:6789/0 2024-03-25T18:19:13.745 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.743+0000 7f5f468af640 10 monclient(hunting): picked mon.noname-a con 0x7f5f401430c0 addr v1:172.21.15.156:6789/0 2024-03-25T18:19:13.746 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.743+0000 7f5f468af640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f40137c60 con 0x7f5f401430c0 2024-03-25T18:19:13.746 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.743+0000 7f5f468af640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f40139160 con 0x7f5f4013f710 2024-03-25T18:19:13.746 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.743+0000 7f5f468af640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f401380f0 con 0x7f5f400a98f0 2024-03-25T18:19:13.746 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.743+0000 7f5f468af640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:13.746 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.743+0000 7f5f468af640 10 monclient(hunting): authenticate will time out at 7635.053914s 2024-03-25T18:19:13.756 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.755+0000 7f5f3ffff640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f5f400a98f0 0x7f5f401dab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:55254/0 (socket says 172.21.15.156:55254) 2024-03-25T18:19:13.756 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.755+0000 7f5f3ffff640 1 -- 172.21.15.156:0/1430929225 learned_addr learned my addr 172.21.15.156:0/1430929225 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:19:13.758 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.757+0000 7f5f3cff9640 10 client.?.objecter ms_handle_connect 0x7f5f400a98f0 2024-03-25T18:19:13.758 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.757+0000 7f5f3cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:13.770 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.770+0000 7f5f3cff9640 10 client.?.objecter ms_handle_connect 0x7f5f4013f710 2024-03-25T18:19:13.771 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.770+0000 7f5f3cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:13.784 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.783+0000 7f5f3cff9640 10 client.?.objecter ms_handle_connect 0x7f5f401430c0 2024-03-25T18:19:13.784 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.783+0000 7f5f3cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:13.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.794+0000 7f5f3cff9640 1 -- 172.21.15.156:0/1430929225 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808229089 0 0) 0x7f5f401380f0 con 0x7f5f400a98f0 2024-03-25T18:19:13.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.794+0000 7f5f3cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:13.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.794+0000 7f5f3cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:13.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.794+0000 7f5f3cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:13.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.794+0000 7f5f3cff9640 10 monclient(hunting): my global_id is 4727 2024-03-25T18:19:13.795 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.794+0000 7f5f3cff9640 1 -- 172.21.15.156:0/1430929225 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f2c004eb0 con 0x7f5f400a98f0 2024-03-25T18:19:13.797 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.796+0000 7f5f3cff9640 1 -- 172.21.15.156:0/1430929225 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102415079 0 0) 0x7f5f40139160 con 0x7f5f4013f710 2024-03-25T18:19:13.797 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.796+0000 7f5f3cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:13.797 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.796+0000 7f5f3cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:13.797 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.796+0000 7f5f3cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:13.798 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.796+0000 7f5f3cff9640 10 monclient(hunting): my global_id is 4723 2024-03-25T18:19:13.798 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.796+0000 7f5f3cff9640 1 -- 172.21.15.156:0/1430929225 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f401380f0 con 0x7f5f4013f710 2024-03-25T18:19:13.808 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.807+0000 7f5f3cff9640 1 -- 172.21.15.156:0/1430929225 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1051039355 0 0) 0x7f5f401380f0 con 0x7f5f4013f710 2024-03-25T18:19:13.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.807+0000 7f5f3cff9640 1 -- 172.21.15.156:0/1430929225 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f40139160 con 0x7f5f4013f710 2024-03-25T18:19:13.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.807+0000 7f5f3cff9640 1 -- 172.21.15.156:0/1430929225 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5f340018b0 con 0x7f5f4013f710 2024-03-25T18:19:13.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.807+0000 7f5f3cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:13.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.807+0000 7f5f3cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:13.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.807+0000 7f5f3cff9640 10 monclient(hunting): dump: 2024-03-25T18:19:13.809 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:19:13.809 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:13.809 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:13.809 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:13.809 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:13.809 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:13.809 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:19:13.809 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:19:13.810 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:19:13.810 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:13.810 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.807+0000 7f5f3cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:13.816 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.814+0000 7f5f3cff9640 1 -- 172.21.15.156:0/1430929225 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3174613978 0 0) 0x7f5f40137c60 con 0x7f5f401430c0 2024-03-25T18:19:13.816 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.814+0000 7f5f3cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:13.816 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.814+0000 7f5f3cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:13.816 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.814+0000 7f5f3cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:13.816 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.814+0000 7f5f3cff9640 10 monclient(hunting): my global_id is 4716 2024-03-25T18:19:13.816 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.815+0000 7f5f3cff9640 1 -- 172.21.15.156:0/1430929225 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f401380f0 con 0x7f5f401430c0 2024-03-25T18:19:13.831 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.829+0000 7f5f3cff9640 1 -- 172.21.15.156:0/1430929225 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 911124650 0 0) 0x7f5f40139160 con 0x7f5f4013f710 2024-03-25T18:19:13.831 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.829+0000 7f5f3cff9640 1 -- 172.21.15.156:0/1430929225 >> v1:172.21.15.156:6790/0 conn(0x7f5f400a98f0 legacy=0x7f5f401dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:13.831 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.829+0000 7f5f3cff9640 1 -- 172.21.15.156:0/1430929225 >> v1:172.21.15.156:6789/0 conn(0x7f5f401430c0 legacy=0x7f5f401e2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:13.831 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.829+0000 7f5f3cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:19:13.831 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.829+0000 7f5f3cff9640 1 monclient: found mon.b 2024-03-25T18:19:13.831 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.829+0000 7f5f3cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:13.831 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.829+0000 7f5f3cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:19:13.831 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.829+0000 7f5f3cff9640 1 -- 172.21.15.156:0/1430929225 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f401397f0 con 0x7f5f4013f710 2024-03-25T18:19:13.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.829+0000 7f5f3cff9640 10 monclient: _finish_auth 0 2024-03-25T18:19:13.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.829+0000 7f5f3cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:19:13.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.829+0000 7f5f3cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:13.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.829+0000 7f5f468af640 5 monclient: authenticate success, global_id 4723 2024-03-25T18:19:13.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.829+0000 7f5f468af640 10 monclient: _renew_subs 2024-03-25T18:19:13.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.829+0000 7f5f468af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:19:13.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.829+0000 7f5f468af640 1 -- 172.21.15.156:0/1430929225 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f401db280 con 0x7f5f4013f710 2024-03-25T18:19:13.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.830+0000 7f5f468af640 10 client.4723.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:13.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.830+0000 7f5f468af640 10 monclient: _renew_subs 2024-03-25T18:19:13.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.830+0000 7f5f468af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:19:13.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.830+0000 7f5f468af640 1 -- 172.21.15.156:0/1430929225 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f40133960 con 0x7f5f4013f710 2024-03-25T18:19:13.832 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.830+0000 7f5f468af640 1 librados: init done 2024-03-25T18:19:13.833 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.831+0000 7f5f468af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:13.833 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.831+0000 7f5f468af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:13.833 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.831+0000 7f5f468af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:19:13.833 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:13.831+0000 7f5f468af640 1 -- 172.21.15.156:0/1430929225 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f40139a40 con 0x7f5f4013f710 2024-03-25T18:19:14.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.127+0000 7f5f3cff9640 1 -- 172.21.15.156:0/1430929225 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f34003610 con 0x7f5f4013f710 2024-03-25T18:19:14.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.127+0000 7f5f3cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:14.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.127+0000 7f5f3e7fc640 4 set_mon_vals no callback set 2024-03-25T18:19:14.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.128+0000 7f5f3cff9640 1 -- 172.21.15.156:0/1430929225 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f5f34003a60 con 0x7f5f4013f710 2024-03-25T18:19:14.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.128+0000 7f5f3cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:14.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.128+0000 7f5f3cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:14.129 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.128+0000 7f5f3cff9640 10 monclient: dump: 2024-03-25T18:19:14.129 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:19:14.130 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:14.130 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:14.130 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:14.130 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:14.130 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:14.130 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:19:14.130 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:19:14.130 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:19:14.130 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:14.131 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.128+0000 7f5f3cff9640 1 -- 172.21.15.156:0/1430929225 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f5f34003c80 con 0x7f5f4013f710 2024-03-25T18:19:14.131 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.129+0000 7f5f3ffff640 10 monclient: get_auth_request con 0x7f5f2c06de10 auth_method 0 2024-03-25T18:19:14.131 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.129+0000 7f5f3cff9640 10 client.4723.objecter ms_handle_connect 0x7f5f2c06de10 2024-03-25T18:19:14.135 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.134+0000 7f5f3cff9640 1 -- 172.21.15.156:0/1430929225 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3381691338 0 0) 0x7f5f340535a0 con 0x7f5f4013f710 2024-03-25T18:19:14.135 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.134+0000 7f5f3cff9640 10 client.4723.objecter ms_dispatch 0x7f5f40000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:19:14.135 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.134+0000 7f5f3cff9640 3 client.4723.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:19:14.135 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.134+0000 7f5f3cff9640 3 client.4723.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:19:14.135 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.134+0000 7f5f3cff9640 20 client.4723.objecter dump_active .. 0 homeless 2024-03-25T18:19:14.135 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.134+0000 7f5f3cff9640 1 -- 172.21.15.156:0/1430929225 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5f3404f0b0 con 0x7f5f4013f710 2024-03-25T18:19:14.135 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.134+0000 7f5f3cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:14.136 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.134+0000 7f5f3cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:14.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.380+0000 7f5f468af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:14.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.380+0000 7f5f468af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:14.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.380+0000 7f5f468af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:19:14.381 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.380+0000 7f5f468af640 1 -- 172.21.15.156:0/1430929225 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f40137e10 con 0x7f5f4013f710 2024-03-25T18:19:14.405 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.404+0000 7f5f3cff9640 1 -- 172.21.15.156:0/1430929225 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1441834839) 0x7f5f40137e10 con 0x7f5f4013f710 2024-03-25T18:19:14.405 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.404+0000 7f5f3cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:14.406 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.404+0000 7f5f3cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:14.406 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:19:14.406 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":11,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:18:20.029906+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:18:20.029906+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:14.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.406+0000 7f5f468af640 10 librados: watch_flush enter 2024-03-25T18:19:14.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.407+0000 7f5f468af640 10 librados: watch_flush exit 2024-03-25T18:19:14.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.407+0000 7f5f468af640 20 client.4723.objecter shutdown clearing up homeless session... 2024-03-25T18:19:14.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.407+0000 7f5f468af640 10 client.4723.objecter successfully canceled tick 2024-03-25T18:19:14.408 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.407+0000 7f5f468af640 1 -- 172.21.15.156:0/1430929225 >> v1:172.21.15.194:6800/2946427713 conn(0x7f5f2c06de10 legacy=0x7f5f2c0702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:14.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.407+0000 7f5f468af640 10 monclient: shutdown 2024-03-25T18:19:14.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.407+0000 7f5f468af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:14.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.407+0000 7f5f468af640 1 -- 172.21.15.156:0/1430929225 >> v1:172.21.15.194:6789/0 conn(0x7f5f4013f710 legacy=0x7f5f40133070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:14.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.407+0000 7f5f468af640 1 -- 172.21.15.156:0/1430929225 shutdown_connections 2024-03-25T18:19:14.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.407+0000 7f5f468af640 1 -- 172.21.15.156:0/1430929225 >> 172.21.15.156:0/1430929225 conn(0x7f5f400b7ea0 msgr2=0x7f5f4013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:14.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.407+0000 7f5f468af640 1 -- 172.21.15.156:0/1430929225 shutdown_connections 2024-03-25T18:19:14.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.407+0000 7f5f468af640 1 -- 172.21.15.156:0/1430929225 wait complete. 2024-03-25T18:19:14.409 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:14.407+0000 7f5f468af640 1 librados: shutdown 2024-03-25T18:19:14.421 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:20.351 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:11:26.341 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:20.355 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:11:26.346 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:20.386 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:11:26.375 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:20.415 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:11:26.381 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:20.419 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:11:26.386 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:20.422 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:11:26.412 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:20.433 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:11:26.423 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:20.434 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:11:26.401 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:20.452 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:11:26.442 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:21.227 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:21.341 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.340+0000 7f095dad3640 10 monclient: get_monmap_and_config 2024-03-25T18:19:21.341 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.340+0000 7f095dad3640 10 monclient: build_initial_monmap 2024-03-25T18:19:21.341 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.340+0000 7f095dad3640 10 monclient: monmap: 2024-03-25T18:19:21.341 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:19:21.341 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:21.341 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:19:21.341246+0000 2024-03-25T18:19:21.342 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:19:21.341246+0000 2024-03-25T18:19:21.342 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:21.342 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:21.342 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:19:21.342 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:19:21.342 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:19:21.342 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:21.343 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.341+0000 7f095dad3640 1 Processor -- start 2024-03-25T18:19:21.343 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.341+0000 7f095dad3640 1 -- start start 2024-03-25T18:19:21.343 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.341+0000 7f095dad3640 10 monclient: init 2024-03-25T18:19:21.343 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.341+0000 7f095dad3640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:21.343 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.341+0000 7f095dad3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:21.344 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.341+0000 7f095dad3640 10 monclient(hunting): picked mon.noname-a con 0x7f09580b4d60 addr v1:172.21.15.156:6789/0 2024-03-25T18:19:21.344 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.341+0000 7f095dad3640 10 monclient(hunting): picked mon.noname-c con 0x7f09580b1730 addr v1:172.21.15.194:6789/0 2024-03-25T18:19:21.344 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.341+0000 7f095dad3640 10 monclient(hunting): picked mon.noname-b con 0x7f095814be90 addr v1:172.21.15.156:6790/0 2024-03-25T18:19:21.344 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.341+0000 7f095dad3640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09580af940 con 0x7f09580b4d60 2024-03-25T18:19:21.344 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.341+0000 7f095dad3640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09580b0a70 con 0x7f09580b1730 2024-03-25T18:19:21.344 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.341+0000 7f095dad3640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09580b0d50 con 0x7f095814be90 2024-03-25T18:19:21.344 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.341+0000 7f095dad3640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:21.344 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.341+0000 7f095dad3640 10 monclient(hunting): authenticate will time out at 7642.652678s 2024-03-25T18:19:21.346 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.345+0000 7f0957fff640 1 --1- >> v1:172.21.15.156:6790/0 conn(0x7f095814be90 0x7f095814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6790/0 says I am v1:172.21.15.156:59812/0 (socket says 172.21.15.156:59812) 2024-03-25T18:19:21.346 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.345+0000 7f0957fff640 1 -- 172.21.15.156:0/891734163 learned_addr learned my addr 172.21.15.156:0/891734163 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:19:21.414 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.413+0000 7f09567fc640 1 -- 172.21.15.156:0/891734163 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 939926763 0 0) 0x7f09580af940 con 0x7f09580b4d60 2024-03-25T18:19:21.414 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.413+0000 7f09567fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:21.414 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.413+0000 7f09567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:21.414 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.413+0000 7f09567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:21.414 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.413+0000 7f09567fc640 10 monclient(hunting): my global_id is 4719 2024-03-25T18:19:21.414 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.413+0000 7f09567fc640 1 -- 172.21.15.156:0/891734163 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0940003520 con 0x7f09580b4d60 2024-03-25T18:19:21.426 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.425+0000 7f09567fc640 1 -- 172.21.15.156:0/891734163 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1761351298 0 0) 0x7f09580b0d50 con 0x7f095814be90 2024-03-25T18:19:21.426 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.425+0000 7f09567fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:21.426 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.425+0000 7f09567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:21.426 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.425+0000 7f09567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:21.426 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.425+0000 7f09567fc640 10 monclient(hunting): my global_id is 4730 2024-03-25T18:19:21.426 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.425+0000 7f09567fc640 1 -- 172.21.15.156:0/891734163 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09580af940 con 0x7f095814be90 2024-03-25T18:19:21.430 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.429+0000 7f09567fc640 1 -- 172.21.15.156:0/891734163 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547696669 0 0) 0x7f09580b0a70 con 0x7f09580b1730 2024-03-25T18:19:21.430 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.429+0000 7f09567fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:21.430 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.429+0000 7f09567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:21.430 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.429+0000 7f09567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:21.431 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.429+0000 7f09567fc640 10 monclient(hunting): my global_id is 4726 2024-03-25T18:19:21.431 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.429+0000 7f09567fc640 1 -- 172.21.15.156:0/891734163 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09580b0d50 con 0x7f09580b1730 2024-03-25T18:19:21.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.430+0000 7f09567fc640 1 -- 172.21.15.156:0/891734163 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2411074087 0 0) 0x7f0940003520 con 0x7f09580b4d60 2024-03-25T18:19:21.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.430+0000 7f09567fc640 1 -- 172.21.15.156:0/891734163 >> v1:172.21.15.156:6790/0 conn(0x7f095814be90 legacy=0x7f095814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:21.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.430+0000 7f09567fc640 1 -- 172.21.15.156:0/891734163 >> v1:172.21.15.194:6789/0 conn(0x7f09580b1730 legacy=0x7f09580b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:21.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.430+0000 7f09567fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:21.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.430+0000 7f09567fc640 1 monclient: found mon.noname-a 2024-03-25T18:19:21.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.430+0000 7f09567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:21.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.430+0000 7f09567fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:19:21.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.430+0000 7f09567fc640 1 -- 172.21.15.156:0/891734163 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09580b1030 con 0x7f09580b4d60 2024-03-25T18:19:21.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.430+0000 7f09567fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:21.432 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.430+0000 7f09567fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:21.433 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.430+0000 7f09567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:21.433 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.430+0000 7f09567fc640 1 -- 172.21.15.156:0/891734163 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f0948001690 con 0x7f09580b4d60 2024-03-25T18:19:21.433 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.430+0000 7f09567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:21.433 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.430+0000 7f09567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:21.433 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.430+0000 7f09567fc640 10 monclient: dump: 2024-03-25T18:19:21.433 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:19:21.433 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:21.433 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:21.433 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:21.433 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:21.433 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:21.433 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:19:21.433 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:19:21.433 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:19:21.433 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:21.433 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.430+0000 7f095dad3640 5 monclient: authenticate success, global_id 4719 2024-03-25T18:19:21.434 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.430+0000 7f095dad3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:21.440 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.439+0000 7f09567fc640 1 -- 172.21.15.156:0/891734163 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09480017f0 con 0x7f09580b4d60 2024-03-25T18:19:21.440 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.439+0000 7f09567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:21.440 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.439+0000 7f09567fc640 1 -- 172.21.15.156:0/891734163 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f0948001c00 con 0x7f09580b4d60 2024-03-25T18:19:21.440 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.439+0000 7f09567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:21.440 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.439+0000 7f09567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:21.440 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.439+0000 7f09567fc640 10 monclient: dump: 2024-03-25T18:19:21.440 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:19:21.440 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:21.440 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:21.440 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:21.440 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:21.440 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:21.440 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:19:21.441 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:19:21.441 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:19:21.441 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:21.441 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.439+0000 7f095dad3640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:21.441 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.439+0000 7f095dad3640 4 set_mon_vals no callback set 2024-03-25T18:19:21.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.439+0000 7f095dad3640 10 monclient: shutdown 2024-03-25T18:19:21.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.439+0000 7f095dad3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:21.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.439+0000 7f095dad3640 1 -- 172.21.15.156:0/891734163 >> v1:172.21.15.156:6789/0 conn(0x7f09580b4d60 legacy=0x7f095813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:21.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.440+0000 7f095dad3640 1 -- 172.21.15.156:0/891734163 shutdown_connections 2024-03-25T18:19:21.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.440+0000 7f095dad3640 1 -- 172.21.15.156:0/891734163 >> 172.21.15.156:0/891734163 conn(0x7f09580617a0 msgr2=0x7f0958061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:21.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.440+0000 7f095dad3640 1 -- 172.21.15.156:0/891734163 shutdown_connections 2024-03-25T18:19:21.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.440+0000 7f095dad3640 1 -- 172.21.15.156:0/891734163 wait complete. 2024-03-25T18:19:21.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.440+0000 7f095dad3640 10 monclient: build_initial_monmap 2024-03-25T18:19:21.443 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.440+0000 7f095dad3640 10 monclient: monmap: 2024-03-25T18:19:21.443 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:19:21.443 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:21.443 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:19:21.443 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:19:21.443 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:21.444 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:19:21.444 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:19:21.444 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:19:21.444 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:19:21.444 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:21.444 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.441+0000 7f095dad3640 1 librados: starting msgr at 2024-03-25T18:19:21.444 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.441+0000 7f095dad3640 1 librados: starting objecter 2024-03-25T18:19:21.444 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.441+0000 7f095dad3640 1 Processor -- start 2024-03-25T18:19:21.444 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.441+0000 7f095dad3640 1 -- start start 2024-03-25T18:19:21.444 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.441+0000 7f095dad3640 1 librados: setting wanted keys 2024-03-25T18:19:21.444 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.441+0000 7f095dad3640 1 librados: calling monclient init 2024-03-25T18:19:21.444 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.441+0000 7f095dad3640 10 monclient: init 2024-03-25T18:19:21.445 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.442+0000 7f095dad3640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:21.445 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.442+0000 7f095dad3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:21.445 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.442+0000 7f095dad3640 10 monclient(hunting): picked mon.noname-c con 0x7f09580b1730 addr v1:172.21.15.156:6790/0 2024-03-25T18:19:21.446 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.442+0000 7f095dad3640 10 monclient(hunting): picked mon.noname-a con 0x7f09580b4d60 addr v1:172.21.15.156:6789/0 2024-03-25T18:19:21.446 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.442+0000 7f095dad3640 10 monclient(hunting): picked mon.noname-b con 0x7f095814be90 addr v1:172.21.15.194:6789/0 2024-03-25T18:19:21.446 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.442+0000 7f095dad3640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0958138210 con 0x7f09580b4d60 2024-03-25T18:19:21.446 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.442+0000 7f095dad3640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09581375a0 con 0x7f095814be90 2024-03-25T18:19:21.446 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.442+0000 7f095dad3640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0958138440 con 0x7f09580b1730 2024-03-25T18:19:21.446 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.442+0000 7f095dad3640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:21.446 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.442+0000 7f095dad3640 10 monclient(hunting): authenticate will time out at 7642.753711s 2024-03-25T18:19:21.457 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.456+0000 7f0957fff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f095814be90 0x7f09581e3280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:47044/0 (socket says 172.21.15.156:47044) 2024-03-25T18:19:21.457 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.456+0000 7f0957fff640 1 -- 172.21.15.156:0/2125751239 learned_addr learned my addr 172.21.15.156:0/2125751239 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:19:21.457 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.456+0000 7f093ffff640 10 client.?.objecter ms_handle_connect 0x7f095814be90 2024-03-25T18:19:21.457 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.456+0000 7f093ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:21.461 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.460+0000 7f093ffff640 10 client.?.objecter ms_handle_connect 0x7f09580b1730 2024-03-25T18:19:21.461 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.460+0000 7f093ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:21.470 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.469+0000 7f093ffff640 10 client.?.objecter ms_handle_connect 0x7f09580b4d60 2024-03-25T18:19:21.470 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.469+0000 7f093ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:21.472 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.471+0000 7f093ffff640 1 -- 172.21.15.156:0/2125751239 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4009417854 0 0) 0x7f0958138440 con 0x7f09580b1730 2024-03-25T18:19:21.472 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.471+0000 7f093ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:21.472 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.471+0000 7f093ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:21.472 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.471+0000 7f093ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:21.472 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.471+0000 7f093ffff640 10 monclient(hunting): my global_id is 4733 2024-03-25T18:19:21.472 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.471+0000 7f093ffff640 1 -- 172.21.15.156:0/2125751239 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0944004eb0 con 0x7f09580b1730 2024-03-25T18:19:21.480 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.479+0000 7f093ffff640 1 -- 172.21.15.156:0/2125751239 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3272682637 0 0) 0x7f09581375a0 con 0x7f095814be90 2024-03-25T18:19:21.480 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.479+0000 7f093ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:21.480 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.479+0000 7f093ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:21.480 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.479+0000 7f093ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:21.480 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.479+0000 7f093ffff640 10 monclient(hunting): my global_id is 4729 2024-03-25T18:19:21.480 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.479+0000 7f093ffff640 1 -- 172.21.15.156:0/2125751239 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0958138440 con 0x7f095814be90 2024-03-25T18:19:21.481 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.480+0000 7f093ffff640 1 -- 172.21.15.156:0/2125751239 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3952611787 0 0) 0x7f0958138210 con 0x7f09580b4d60 2024-03-25T18:19:21.482 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.480+0000 7f093ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:21.482 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.480+0000 7f093ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:21.482 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.480+0000 7f093ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:21.482 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.480+0000 7f093ffff640 10 monclient(hunting): my global_id is 4722 2024-03-25T18:19:21.482 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.480+0000 7f093ffff640 1 -- 172.21.15.156:0/2125751239 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09581375a0 con 0x7f09580b4d60 2024-03-25T18:19:21.491 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.489+0000 7f093ffff640 1 -- 172.21.15.156:0/2125751239 <== mon.2 v1:172.21.15.156:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2884273548 0 0) 0x7f0944004eb0 con 0x7f09580b1730 2024-03-25T18:19:21.491 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.489+0000 7f093ffff640 1 -- 172.21.15.156:0/2125751239 --> v1:172.21.15.156:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0958138210 con 0x7f09580b1730 2024-03-25T18:19:21.491 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.489+0000 7f093ffff640 1 -- 172.21.15.156:0/2125751239 <== mon.2 v1:172.21.15.156:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f0948001b00 con 0x7f09580b1730 2024-03-25T18:19:21.491 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.489+0000 7f093ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:21.491 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.489+0000 7f093ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:21.491 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.489+0000 7f093ffff640 10 monclient(hunting): dump: 2024-03-25T18:19:21.491 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:19:21.491 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:21.491 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:21.491 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:21.491 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:21.491 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:21.491 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:19:21.492 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:19:21.492 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:19:21.492 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:21.492 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.489+0000 7f093ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:21.495 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.494+0000 7f093ffff640 1 -- 172.21.15.156:0/2125751239 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1955067774 0 0) 0x7f0958138440 con 0x7f095814be90 2024-03-25T18:19:21.495 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.494+0000 7f093ffff640 1 -- 172.21.15.156:0/2125751239 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0944004eb0 con 0x7f095814be90 2024-03-25T18:19:21.495 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.494+0000 7f093ffff640 1 -- 172.21.15.156:0/2125751239 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f0950001c50 con 0x7f095814be90 2024-03-25T18:19:21.495 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.494+0000 7f093ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:21.495 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.494+0000 7f093ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:21.495 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.494+0000 7f093ffff640 10 monclient(hunting): dump: 2024-03-25T18:19:21.495 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:19:21.495 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:21.496 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:21.496 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:21.496 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:21.496 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:21.496 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:19:21.496 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:19:21.496 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:19:21.496 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:21.497 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.496+0000 7f093ffff640 1 -- 172.21.15.156:0/2125751239 <== mon.2 v1:172.21.15.156:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2605308097 0 0) 0x7f0958138210 con 0x7f09580b1730 2024-03-25T18:19:21.497 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.496+0000 7f093ffff640 1 -- 172.21.15.156:0/2125751239 >> v1:172.21.15.194:6789/0 conn(0x7f095814be90 legacy=0x7f09581e3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:21.497 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.496+0000 7f093ffff640 1 -- 172.21.15.156:0/2125751239 >> v1:172.21.15.156:6789/0 conn(0x7f09580b4d60 legacy=0x7f095812fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:21.497 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.496+0000 7f093ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:21.498 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.496+0000 7f093ffff640 1 monclient: found mon.c 2024-03-25T18:19:21.498 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.496+0000 7f093ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:21.498 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.496+0000 7f093ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:19:21.498 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.496+0000 7f093ffff640 1 -- 172.21.15.156:0/2125751239 --> v1:172.21.15.156:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09581364b0 con 0x7f09580b1730 2024-03-25T18:19:21.498 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.496+0000 7f093ffff640 10 monclient: _finish_auth 0 2024-03-25T18:19:21.498 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.496+0000 7f093ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:21.498 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.496+0000 7f093ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:21.498 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.496+0000 7f095dad3640 5 monclient: authenticate success, global_id 4733 2024-03-25T18:19:21.498 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.496+0000 7f095dad3640 10 monclient: _renew_subs 2024-03-25T18:19:21.498 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.496+0000 7f095dad3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:19:21.498 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.496+0000 7f095dad3640 1 -- 172.21.15.156:0/2125751239 --> v1:172.21.15.156:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f095813b470 con 0x7f09580b1730 2024-03-25T18:19:21.498 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.496+0000 7f095dad3640 10 client.4733.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:21.498 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.496+0000 7f095dad3640 10 monclient: _renew_subs 2024-03-25T18:19:21.498 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.496+0000 7f095dad3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:19:21.498 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.496+0000 7f095dad3640 1 -- 172.21.15.156:0/2125751239 --> v1:172.21.15.156:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0958136230 con 0x7f09580b1730 2024-03-25T18:19:21.499 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.496+0000 7f095dad3640 1 librados: init done 2024-03-25T18:19:21.499 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.497+0000 7f095dad3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:21.499 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.497+0000 7f095dad3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:21.499 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.497+0000 7f095dad3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:19:21.499 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.497+0000 7f095dad3640 1 -- 172.21.15.156:0/2125751239 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09580627b0 con 0x7f09580b1730 2024-03-25T18:19:21.520 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.519+0000 7f093ffff640 1 -- 172.21.15.156:0/2125751239 <== mon.2 v1:172.21.15.156:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09480038a0 con 0x7f09580b1730 2024-03-25T18:19:21.520 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.519+0000 7f093ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:21.520 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.519+0000 7f093ffff640 1 -- 172.21.15.156:0/2125751239 <== mon.2 v1:172.21.15.156:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f0948003bb0 con 0x7f09580b1730 2024-03-25T18:19:21.520 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.519+0000 7f093ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:21.520 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.519+0000 7f093ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:21.520 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.519+0000 7f093ffff640 10 monclient: dump: 2024-03-25T18:19:21.520 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:19:21.520 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:21.520 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:21.521 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:21.521 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:21.521 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:21.521 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:19:21.521 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:19:21.521 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:19:21.521 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:21.521 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.519+0000 7f0955ffb640 4 set_mon_vals no callback set 2024-03-25T18:19:21.522 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.520+0000 7f093ffff640 1 -- 172.21.15.156:0/2125751239 <== mon.2 v1:172.21.15.156:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f0948019b80 con 0x7f09580b1730 2024-03-25T18:19:21.522 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.520+0000 7f0956ffd640 10 monclient: get_auth_request con 0x7f094406df70 auth_method 0 2024-03-25T18:19:21.522 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.521+0000 7f093ffff640 10 client.4733.objecter ms_handle_connect 0x7f094406df70 2024-03-25T18:19:21.524 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.523+0000 7f093ffff640 1 -- 172.21.15.156:0/2125751239 <== mon.2 v1:172.21.15.156:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3381691338 0 0) 0x7f0948053630 con 0x7f09580b1730 2024-03-25T18:19:21.524 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.523+0000 7f093ffff640 10 client.4733.objecter ms_dispatch 0x7f0958000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:19:21.524 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.523+0000 7f093ffff640 3 client.4733.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:19:21.525 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.523+0000 7f093ffff640 3 client.4733.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:19:21.525 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.523+0000 7f093ffff640 20 client.4733.objecter dump_active .. 0 homeless 2024-03-25T18:19:21.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.808+0000 7f093ffff640 1 -- 172.21.15.156:0/2125751239 <== mon.2 v1:172.21.15.156:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f094804f140 con 0x7f09580b1730 2024-03-25T18:19:21.809 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.808+0000 7f093ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:21.810 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:21.808+0000 7f093ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:22.066 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:22.065+0000 7f095dad3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:22.066 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:22.065+0000 7f095dad3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:22.066 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:22.065+0000 7f095dad3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.156:6790/0 2024-03-25T18:19:22.067 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:22.065+0000 7f095dad3640 1 -- 172.21.15.156:0/2125751239 --> v1:172.21.15.156:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0958136640 con 0x7f09580b1730 2024-03-25T18:19:22.098 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:22.097+0000 7f093ffff640 1 -- 172.21.15.156:0/2125751239 <== mon.2 v1:172.21.15.156:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1441834839) 0x7f0948051070 con 0x7f09580b1730 2024-03-25T18:19:22.099 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:22.097+0000 7f093ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:22.099 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:22.097+0000 7f093ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:22.099 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:19:22.099 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":11,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:18:20.029906+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:18:20.029906+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:22.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:22.099+0000 7f095dad3640 10 librados: watch_flush enter 2024-03-25T18:19:22.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:22.099+0000 7f095dad3640 10 librados: watch_flush exit 2024-03-25T18:19:22.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:22.099+0000 7f095dad3640 20 client.4733.objecter shutdown clearing up homeless session... 2024-03-25T18:19:22.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:22.099+0000 7f095dad3640 10 client.4733.objecter successfully canceled tick 2024-03-25T18:19:22.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:22.099+0000 7f095dad3640 1 -- 172.21.15.156:0/2125751239 >> v1:172.21.15.194:6800/2946427713 conn(0x7f094406df70 legacy=0x7f0944070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:22.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:22.099+0000 7f095dad3640 10 monclient: shutdown 2024-03-25T18:19:22.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:22.099+0000 7f095dad3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:22.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:22.099+0000 7f095dad3640 1 -- 172.21.15.156:0/2125751239 >> v1:172.21.15.156:6790/0 conn(0x7f09580b1730 legacy=0x7f0958139320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:22.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:22.099+0000 7f095dad3640 1 -- 172.21.15.156:0/2125751239 shutdown_connections 2024-03-25T18:19:22.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:22.099+0000 7f095dad3640 1 -- 172.21.15.156:0/2125751239 >> 172.21.15.156:0/2125751239 conn(0x7f09580617a0 msgr2=0x7f095813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:22.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:22.100+0000 7f095dad3640 1 -- 172.21.15.156:0/2125751239 shutdown_connections 2024-03-25T18:19:22.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:22.100+0000 7f095dad3640 1 -- 172.21.15.156:0/2125751239 wait complete. 2024-03-25T18:19:22.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:22.100+0000 7f095dad3640 1 librados: shutdown 2024-03-25T18:19:22.113 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:28.918 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:29.034 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.033+0000 7f226dc06640 10 monclient: get_monmap_and_config 2024-03-25T18:19:29.034 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.033+0000 7f226dc06640 10 monclient: build_initial_monmap 2024-03-25T18:19:29.034 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.033+0000 7f226dc06640 10 monclient: monmap: 2024-03-25T18:19:29.034 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:19:29.034 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:29.035 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:19:29.034612+0000 2024-03-25T18:19:29.035 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:19:29.034612+0000 2024-03-25T18:19:29.035 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:29.035 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:29.035 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:19:29.035 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:19:29.035 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:19:29.035 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:29.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.034+0000 7f226dc06640 1 Processor -- start 2024-03-25T18:19:29.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.034+0000 7f226dc06640 1 -- start start 2024-03-25T18:19:29.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.034+0000 7f226dc06640 10 monclient: init 2024-03-25T18:19:29.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.035+0000 7f226dc06640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:29.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.035+0000 7f226dc06640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:29.036 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.035+0000 7f226dc06640 10 monclient(hunting): picked mon.noname-c con 0x7f22680b4d60 addr v1:172.21.15.194:6789/0 2024-03-25T18:19:29.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.035+0000 7f226dc06640 10 monclient(hunting): picked mon.noname-b con 0x7f22680b1730 addr v1:172.21.15.156:6790/0 2024-03-25T18:19:29.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.035+0000 7f226dc06640 10 monclient(hunting): picked mon.noname-a con 0x7f226814be90 addr v1:172.21.15.156:6789/0 2024-03-25T18:19:29.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.035+0000 7f226dc06640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22680af940 con 0x7f226814be90 2024-03-25T18:19:29.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.035+0000 7f226dc06640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22680b0a70 con 0x7f22680b4d60 2024-03-25T18:19:29.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.035+0000 7f226dc06640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22680b0d50 con 0x7f22680b1730 2024-03-25T18:19:29.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.035+0000 7f226dc06640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:29.037 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.035+0000 7f226dc06640 10 monclient(hunting): authenticate will time out at 7650.346233s 2024-03-25T18:19:29.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.042+0000 7f2267fff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f226814be90 0x7f226814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:48122/0 (socket says 172.21.15.156:48122) 2024-03-25T18:19:29.043 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.042+0000 7f2267fff640 1 -- 172.21.15.156:0/3592285181 learned_addr learned my addr 172.21.15.156:0/3592285181 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:19:29.055 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.054+0000 7f22667fc640 1 -- 172.21.15.156:0/3592285181 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101897494 0 0) 0x7f22680af940 con 0x7f226814be90 2024-03-25T18:19:29.055 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.054+0000 7f22667fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:29.055 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.054+0000 7f22667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:29.055 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.054+0000 7f22667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:29.055 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.054+0000 7f22667fc640 10 monclient(hunting): my global_id is 4725 2024-03-25T18:19:29.055 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.054+0000 7f22667fc640 1 -- 172.21.15.156:0/3592285181 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2250003520 con 0x7f226814be90 2024-03-25T18:19:29.063 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.062+0000 7f22667fc640 1 -- 172.21.15.156:0/3592285181 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1771428729 0 0) 0x7f2250003520 con 0x7f226814be90 2024-03-25T18:19:29.063 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.062+0000 7f22667fc640 1 -- 172.21.15.156:0/3592285181 >> v1:172.21.15.156:6790/0 conn(0x7f22680b1730 legacy=0x7f22680b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:19:29.063 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.062+0000 7f22667fc640 1 -- 172.21.15.156:0/3592285181 >> v1:172.21.15.194:6789/0 conn(0x7f22680b4d60 legacy=0x7f226813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:29.063 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.062+0000 7f22667fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:29.063 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.062+0000 7f22667fc640 1 monclient: found mon.noname-a 2024-03-25T18:19:29.063 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.062+0000 7f22667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:29.063 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.062+0000 7f22667fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:19:29.063 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.062+0000 7f22667fc640 1 -- 172.21.15.156:0/3592285181 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22680b1030 con 0x7f226814be90 2024-03-25T18:19:29.063 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.062+0000 7f22667fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:29.064 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.062+0000 7f22667fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:29.064 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.062+0000 7f22667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:29.064 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.062+0000 7f22667fc640 1 -- 172.21.15.156:0/3592285181 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f2260001a10 con 0x7f226814be90 2024-03-25T18:19:29.064 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.062+0000 7f22667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:29.064 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.062+0000 7f22667fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:29.064 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.062+0000 7f22667fc640 10 monclient: dump: 2024-03-25T18:19:29.064 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:19:29.064 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:29.064 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:29.064 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:29.064 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:29.064 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:29.064 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:19:29.064 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:19:29.064 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:19:29.065 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:29.065 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.062+0000 7f226dc06640 5 monclient: authenticate success, global_id 4725 2024-03-25T18:19:29.065 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.062+0000 7f226dc06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:29.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.075+0000 7f22667fc640 1 -- 172.21.15.156:0/3592285181 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2260001be0 con 0x7f226814be90 2024-03-25T18:19:29.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.075+0000 7f22667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:29.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.075+0000 7f22667fc640 1 -- 172.21.15.156:0/3592285181 <== mon.0 v1:172.21.15.156:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f2260002030 con 0x7f226814be90 2024-03-25T18:19:29.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.075+0000 7f22667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:29.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.075+0000 7f22667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:29.076 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.075+0000 7f22667fc640 10 monclient: dump: 2024-03-25T18:19:29.076 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:19:29.076 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:29.076 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:29.076 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:29.076 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:29.076 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:29.077 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:19:29.077 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:19:29.077 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:19:29.077 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:29.077 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.075+0000 7f226dc06640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:29.077 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.075+0000 7f226dc06640 4 set_mon_vals no callback set 2024-03-25T18:19:29.078 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.075+0000 7f226dc06640 10 monclient: shutdown 2024-03-25T18:19:29.079 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.075+0000 7f226dc06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:29.079 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.075+0000 7f226dc06640 1 -- 172.21.15.156:0/3592285181 >> v1:172.21.15.156:6789/0 conn(0x7f226814be90 legacy=0x7f226814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:29.079 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.075+0000 7f226dc06640 1 -- 172.21.15.156:0/3592285181 shutdown_connections 2024-03-25T18:19:29.079 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.075+0000 7f226dc06640 1 -- 172.21.15.156:0/3592285181 >> 172.21.15.156:0/3592285181 conn(0x7f22680617a0 msgr2=0x7f2268061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:29.079 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.076+0000 7f226dc06640 1 -- 172.21.15.156:0/3592285181 shutdown_connections 2024-03-25T18:19:29.079 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.076+0000 7f226dc06640 1 -- 172.21.15.156:0/3592285181 wait complete. 2024-03-25T18:19:29.079 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.076+0000 7f226dc06640 10 monclient: build_initial_monmap 2024-03-25T18:19:29.079 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.076+0000 7f226dc06640 10 monclient: monmap: 2024-03-25T18:19:29.079 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:19:29.079 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:29.079 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:19:29.079 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:19:29.079 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:29.079 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:19:29.079 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:19:29.080 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:19:29.080 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:19:29.080 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:29.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.077+0000 7f226dc06640 1 librados: starting msgr at 2024-03-25T18:19:29.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.077+0000 7f226dc06640 1 librados: starting objecter 2024-03-25T18:19:29.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.077+0000 7f226dc06640 1 Processor -- start 2024-03-25T18:19:29.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.077+0000 7f226dc06640 1 -- start start 2024-03-25T18:19:29.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.077+0000 7f226dc06640 1 librados: setting wanted keys 2024-03-25T18:19:29.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.077+0000 7f226dc06640 1 librados: calling monclient init 2024-03-25T18:19:29.080 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.077+0000 7f226dc06640 10 monclient: init 2024-03-25T18:19:29.081 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.078+0000 7f226dc06640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:29.081 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.078+0000 7f226dc06640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:29.081 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.078+0000 7f226dc06640 10 monclient(hunting): picked mon.noname-b con 0x7f22680b1730 addr v1:172.21.15.194:6789/0 2024-03-25T18:19:29.081 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.078+0000 7f226dc06640 10 monclient(hunting): picked mon.noname-a con 0x7f22680b4d60 addr v1:172.21.15.156:6789/0 2024-03-25T18:19:29.081 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.078+0000 7f226dc06640 10 monclient(hunting): picked mon.noname-c con 0x7f226814be90 addr v1:172.21.15.156:6790/0 2024-03-25T18:19:29.081 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.078+0000 7f226dc06640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22681d79c0 con 0x7f22680b4d60 2024-03-25T18:19:29.081 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.078+0000 7f226dc06640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22681d7690 con 0x7f22680b1730 2024-03-25T18:19:29.082 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.078+0000 7f226dc06640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22681d8560 con 0x7f226814be90 2024-03-25T18:19:29.082 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.078+0000 7f226dc06640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:29.082 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.078+0000 7f226dc06640 10 monclient(hunting): authenticate will time out at 7650.389595s 2024-03-25T18:19:29.083 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.082+0000 7f22677fe640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f22680b1730 0x7f22681d7e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:55026/0 (socket says 172.21.15.156:55026) 2024-03-25T18:19:29.083 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.082+0000 7f22677fe640 1 -- 172.21.15.156:0/2731316642 learned_addr learned my addr 172.21.15.156:0/2731316642 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:19:29.084 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.083+0000 7f224ffff640 10 client.?.objecter ms_handle_connect 0x7f22680b1730 2024-03-25T18:19:29.084 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.083+0000 7f224ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:29.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.087+0000 7f224ffff640 1 -- 172.21.15.156:0/2731316642 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656186779 0 0) 0x7f22681d7690 con 0x7f22680b1730 2024-03-25T18:19:29.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.087+0000 7f224ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:29.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.087+0000 7f224ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:29.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.087+0000 7f224ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:29.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.087+0000 7f224ffff640 10 monclient(hunting): my global_id is 4735 2024-03-25T18:19:29.088 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.087+0000 7f224ffff640 1 -- 172.21.15.156:0/2731316642 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2254004eb0 con 0x7f22680b1730 2024-03-25T18:19:29.092 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.091+0000 7f224ffff640 10 client.?.objecter ms_handle_connect 0x7f226814be90 2024-03-25T18:19:29.092 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.091+0000 7f224ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:29.096 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.095+0000 7f224ffff640 10 client.?.objecter ms_handle_connect 0x7f22680b4d60 2024-03-25T18:19:29.096 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.095+0000 7f224ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:29.099 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.098+0000 7f224ffff640 1 -- 172.21.15.156:0/2731316642 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568252189 0 0) 0x7f22681d79c0 con 0x7f22680b4d60 2024-03-25T18:19:29.100 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.098+0000 7f224ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:29.100 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.098+0000 7f224ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:29.100 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.098+0000 7f224ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:29.100 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.098+0000 7f224ffff640 10 monclient(hunting): my global_id is 4728 2024-03-25T18:19:29.100 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.098+0000 7f224ffff640 1 -- 172.21.15.156:0/2731316642 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22681d7690 con 0x7f22680b4d60 2024-03-25T18:19:29.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.100+0000 7f224ffff640 1 -- 172.21.15.156:0/2731316642 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373851642 0 0) 0x7f22681d8560 con 0x7f226814be90 2024-03-25T18:19:29.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.100+0000 7f224ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:29.101 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.100+0000 7f224ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:29.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.100+0000 7f224ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:29.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.100+0000 7f224ffff640 10 monclient(hunting): my global_id is 4736 2024-03-25T18:19:29.102 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.100+0000 7f224ffff640 1 -- 172.21.15.156:0/2731316642 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22681d79c0 con 0x7f226814be90 2024-03-25T18:19:29.104 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.102+0000 7f224ffff640 1 -- 172.21.15.156:0/2731316642 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2578552678 0 0) 0x7f2254004eb0 con 0x7f22680b1730 2024-03-25T18:19:29.104 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.102+0000 7f224ffff640 1 -- 172.21.15.156:0/2731316642 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22681d8560 con 0x7f22680b1730 2024-03-25T18:19:29.104 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.102+0000 7f224ffff640 1 -- 172.21.15.156:0/2731316642 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f2258001a30 con 0x7f22680b1730 2024-03-25T18:19:29.104 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.102+0000 7f224ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:29.104 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.102+0000 7f224ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:29.104 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.102+0000 7f224ffff640 10 monclient(hunting): dump: 2024-03-25T18:19:29.104 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:19:29.104 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:29.104 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:29.104 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:29.104 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:29.104 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:29.104 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:19:29.104 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:19:29.104 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:19:29.105 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:29.105 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.102+0000 7f224ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:29.114 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.113+0000 7f224ffff640 1 -- 172.21.15.156:0/2731316642 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 337883407 0 0) 0x7f22681d8560 con 0x7f22680b1730 2024-03-25T18:19:29.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.113+0000 7f224ffff640 1 -- 172.21.15.156:0/2731316642 >> v1:172.21.15.156:6790/0 conn(0x7f226814be90 legacy=0x7f22681e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:29.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.113+0000 7f224ffff640 1 -- 172.21.15.156:0/2731316642 >> v1:172.21.15.156:6789/0 conn(0x7f22680b4d60 legacy=0x7f2268137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:29.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.113+0000 7f224ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:29.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.113+0000 7f224ffff640 1 monclient: found mon.b 2024-03-25T18:19:29.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.113+0000 7f224ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:29.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.113+0000 7f224ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:19:29.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.113+0000 7f224ffff640 1 -- 172.21.15.156:0/2731316642 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2268142e30 con 0x7f22680b1730 2024-03-25T18:19:29.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.113+0000 7f224ffff640 10 monclient: _finish_auth 0 2024-03-25T18:19:29.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.113+0000 7f224ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:29.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.113+0000 7f224ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:29.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.113+0000 7f226dc06640 5 monclient: authenticate success, global_id 4735 2024-03-25T18:19:29.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.113+0000 7f226dc06640 10 monclient: _renew_subs 2024-03-25T18:19:29.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.113+0000 7f226dc06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:19:29.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.113+0000 7f226dc06640 1 -- 172.21.15.156:0/2731316642 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2268143080 con 0x7f22680b1730 2024-03-25T18:19:29.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.113+0000 7f226dc06640 10 client.4735.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:29.115 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.113+0000 7f226dc06640 10 monclient: _renew_subs 2024-03-25T18:19:29.116 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.113+0000 7f226dc06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:19:29.116 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.113+0000 7f226dc06640 1 -- 172.21.15.156:0/2731316642 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2268142b50 con 0x7f22680b1730 2024-03-25T18:19:29.116 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.113+0000 7f226dc06640 1 librados: init done 2024-03-25T18:19:29.116 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.114+0000 7f226dc06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:29.116 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.114+0000 7f226dc06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:29.116 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.114+0000 7f226dc06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:19:29.116 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.114+0000 7f226dc06640 1 -- 172.21.15.156:0/2731316642 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22680627b0 con 0x7f22680b1730 2024-03-25T18:19:29.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.397+0000 7f224ffff640 1 -- 172.21.15.156:0/2731316642 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2258001c70 con 0x7f22680b1730 2024-03-25T18:19:29.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.397+0000 7f224ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:29.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.397+0000 7f224ffff640 1 -- 172.21.15.156:0/2731316642 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f2258003a60 con 0x7f22680b1730 2024-03-25T18:19:29.398 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.397+0000 7f224ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:29.399 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.397+0000 7f224ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:29.399 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.397+0000 7f224ffff640 10 monclient: dump: 2024-03-25T18:19:29.399 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:19:29.399 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:29.399 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:29.399 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:29.399 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:29.399 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:29.399 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:19:29.399 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:19:29.399 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:19:29.399 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:29.399 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.397+0000 7f2265ffb640 4 set_mon_vals no callback set 2024-03-25T18:19:29.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.398+0000 7f224ffff640 1 -- 172.21.15.156:0/2731316642 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f2258003c40 con 0x7f22680b1730 2024-03-25T18:19:29.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.398+0000 7f2266ffd640 10 monclient: get_auth_request con 0x7f225406de20 auth_method 0 2024-03-25T18:19:29.400 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.398+0000 7f224ffff640 10 client.4735.objecter ms_handle_connect 0x7f225406de20 2024-03-25T18:19:29.403 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.402+0000 7f224ffff640 1 -- 172.21.15.156:0/2731316642 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3381691338 0 0) 0x7f22580548b0 con 0x7f22680b1730 2024-03-25T18:19:29.403 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.402+0000 7f224ffff640 10 client.4735.objecter ms_dispatch 0x7f2268000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:19:29.403 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.402+0000 7f224ffff640 3 client.4735.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:19:29.403 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.402+0000 7f224ffff640 3 client.4735.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:19:29.403 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.402+0000 7f224ffff640 20 client.4735.objecter dump_active .. 0 homeless 2024-03-25T18:19:29.404 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.403+0000 7f224ffff640 1 -- 172.21.15.156:0/2731316642 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f22580503c0 con 0x7f22680b1730 2024-03-25T18:19:29.404 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.403+0000 7f224ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:29.404 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.403+0000 7f224ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:29.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.631+0000 7f226dc06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:29.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.631+0000 7f226dc06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:29.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.631+0000 7f226dc06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:19:29.632 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.631+0000 7f226dc06640 1 -- 172.21.15.156:0/2731316642 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2268069f40 con 0x7f22680b1730 2024-03-25T18:19:29.660 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.659+0000 7f224ffff640 1 -- 172.21.15.156:0/2731316642 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1441834839) 0x7f22580522f0 con 0x7f22680b1730 2024-03-25T18:19:29.660 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.659+0000 7f224ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:29.660 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.659+0000 7f224ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:29.660 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:19:29.660 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":11,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:18:20.029906+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:18:20.029906+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:29.663 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.661+0000 7f226dc06640 10 librados: watch_flush enter 2024-03-25T18:19:29.663 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.661+0000 7f226dc06640 10 librados: watch_flush exit 2024-03-25T18:19:29.663 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.661+0000 7f226dc06640 20 client.4735.objecter shutdown clearing up homeless session... 2024-03-25T18:19:29.663 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.661+0000 7f226dc06640 10 client.4735.objecter successfully canceled tick 2024-03-25T18:19:29.663 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.661+0000 7f226dc06640 1 -- 172.21.15.156:0/2731316642 >> v1:172.21.15.194:6800/2946427713 conn(0x7f225406de20 legacy=0x7f22540702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:29.663 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.661+0000 7f226dc06640 10 monclient: shutdown 2024-03-25T18:19:29.663 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.661+0000 7f226dc06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:29.663 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.661+0000 7f226dc06640 1 -- 172.21.15.156:0/2731316642 >> v1:172.21.15.194:6789/0 conn(0x7f22680b1730 legacy=0x7f22681d7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:29.663 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.661+0000 7f226dc06640 1 -- 172.21.15.156:0/2731316642 shutdown_connections 2024-03-25T18:19:29.663 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.661+0000 7f226dc06640 1 -- 172.21.15.156:0/2731316642 >> 172.21.15.156:0/2731316642 conn(0x7f22680617a0 msgr2=0x7f226813c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:29.663 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.661+0000 7f226dc06640 1 -- 172.21.15.156:0/2731316642 shutdown_connections 2024-03-25T18:19:29.663 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.661+0000 7f226dc06640 1 -- 172.21.15.156:0/2731316642 wait complete. 2024-03-25T18:19:29.664 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:29.662+0000 7f226dc06640 1 librados: shutdown 2024-03-25T18:19:29.674 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:36.481 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:36.599 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.597+0000 7f667a9d4640 10 monclient: get_monmap_and_config 2024-03-25T18:19:36.599 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.597+0000 7f667a9d4640 10 monclient: build_initial_monmap 2024-03-25T18:19:36.599 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.597+0000 7f667a9d4640 10 monclient: monmap: 2024-03-25T18:19:36.599 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:19:36.599 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:36.599 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:19:36.599332+0000 2024-03-25T18:19:36.599 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:19:36.599332+0000 2024-03-25T18:19:36.600 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:36.600 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:36.600 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:19:36.600 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:19:36.600 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:19:36.600 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:36.601 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.598+0000 7f667a9d4640 1 Processor -- start 2024-03-25T18:19:36.601 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.598+0000 7f667a9d4640 1 -- start start 2024-03-25T18:19:36.601 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.598+0000 7f667a9d4640 10 monclient: init 2024-03-25T18:19:36.602 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.598+0000 7f667a9d4640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:36.602 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.598+0000 7f667a9d4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:36.602 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.598+0000 7f667a9d4640 10 monclient(hunting): picked mon.noname-c con 0x7f66740b4d60 addr v1:172.21.15.194:6789/0 2024-03-25T18:19:36.602 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.598+0000 7f667a9d4640 10 monclient(hunting): picked mon.noname-b con 0x7f66740b1730 addr v1:172.21.15.156:6790/0 2024-03-25T18:19:36.602 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.598+0000 7f667a9d4640 10 monclient(hunting): picked mon.noname-a con 0x7f667414be90 addr v1:172.21.15.156:6789/0 2024-03-25T18:19:36.602 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.598+0000 7f667a9d4640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66740af940 con 0x7f667414be90 2024-03-25T18:19:36.602 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.598+0000 7f667a9d4640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66740b0a70 con 0x7f66740b4d60 2024-03-25T18:19:36.602 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.598+0000 7f667a9d4640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66740b0d50 con 0x7f66740b1730 2024-03-25T18:19:36.602 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.598+0000 7f667a9d4640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:36.602 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.598+0000 7f667a9d4640 10 monclient(hunting): authenticate will time out at 7657.910783s 2024-03-25T18:19:36.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.606+0000 7f6673fff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f66740b4d60 0x7f667413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:49078/0 (socket says 172.21.15.156:49078) 2024-03-25T18:19:36.608 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.606+0000 7f6673fff640 1 -- 172.21.15.156:0/3867574946 learned_addr learned my addr 172.21.15.156:0/3867574946 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:19:36.619 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.616+0000 7f6672ffd640 1 -- 172.21.15.156:0/3867574946 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565324322 0 0) 0x7f66740b0a70 con 0x7f66740b4d60 2024-03-25T18:19:36.619 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.617+0000 7f6672ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:36.619 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.617+0000 7f6672ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:36.619 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.617+0000 7f6672ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:36.619 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.617+0000 7f6672ffd640 10 monclient(hunting): my global_id is 4738 2024-03-25T18:19:36.619 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.617+0000 7f6672ffd640 1 -- 172.21.15.156:0/3867574946 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f665c003520 con 0x7f66740b4d60 2024-03-25T18:19:36.627 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.625+0000 7f6672ffd640 1 -- 172.21.15.156:0/3867574946 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2892359400 0 0) 0x7f66740b0d50 con 0x7f66740b1730 2024-03-25T18:19:36.627 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.625+0000 7f6672ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:36.627 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.625+0000 7f6672ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:36.627 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.625+0000 7f6672ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:36.627 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.625+0000 7f6672ffd640 10 monclient(hunting): my global_id is 4739 2024-03-25T18:19:36.627 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.625+0000 7f6672ffd640 1 -- 172.21.15.156:0/3867574946 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66740b0a70 con 0x7f66740b1730 2024-03-25T18:19:36.634 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.632+0000 7f6672ffd640 1 -- 172.21.15.156:0/3867574946 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1745794554 0 0) 0x7f665c003520 con 0x7f66740b4d60 2024-03-25T18:19:36.635 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.632+0000 7f6672ffd640 1 -- 172.21.15.156:0/3867574946 >> v1:172.21.15.156:6790/0 conn(0x7f66740b1730 legacy=0x7f66740b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:36.635 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.632+0000 7f6672ffd640 1 -- 172.21.15.156:0/3867574946 >> v1:172.21.15.156:6789/0 conn(0x7f667414be90 legacy=0x7f667414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:36.635 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.632+0000 7f6672ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:36.635 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.632+0000 7f6672ffd640 1 monclient: found mon.noname-c 2024-03-25T18:19:36.635 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.632+0000 7f6672ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:36.635 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.632+0000 7f6672ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-25T18:19:36.635 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.632+0000 7f6672ffd640 1 -- 172.21.15.156:0/3867574946 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66740b1030 con 0x7f66740b4d60 2024-03-25T18:19:36.635 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.632+0000 7f6672ffd640 10 monclient: _finish_auth 0 2024-03-25T18:19:36.635 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.632+0000 7f6672ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:36.635 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.632+0000 7f6672ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:36.635 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.632+0000 7f6672ffd640 1 -- 172.21.15.156:0/3867574946 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f6664001690 con 0x7f66740b4d60 2024-03-25T18:19:36.635 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.632+0000 7f667a9d4640 5 monclient: authenticate success, global_id 4738 2024-03-25T18:19:36.635 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.632+0000 7f667a9d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:36.635 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.632+0000 7f6672ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:36.636 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.632+0000 7f6672ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:36.636 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.632+0000 7f6672ffd640 10 monclient: dump: 2024-03-25T18:19:36.636 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:19:36.636 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:36.636 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:36.636 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:36.636 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:36.636 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:36.636 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:19:36.636 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:19:36.636 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:19:36.636 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:36.636 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.632+0000 7f667a9d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:36.653 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.651+0000 7f6672ffd640 1 -- 172.21.15.156:0/3867574946 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66640017f0 con 0x7f66740b4d60 2024-03-25T18:19:36.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.651+0000 7f6672ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:36.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.651+0000 7f6672ffd640 1 -- 172.21.15.156:0/3867574946 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f6664001c00 con 0x7f66740b4d60 2024-03-25T18:19:36.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.651+0000 7f667a9d4640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:36.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.651+0000 7f667a9d4640 4 set_mon_vals no callback set 2024-03-25T18:19:36.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.651+0000 7f6672ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:36.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.651+0000 7f6672ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:36.654 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.651+0000 7f6672ffd640 10 monclient: dump: 2024-03-25T18:19:36.654 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:19:36.654 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:36.654 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:36.654 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:36.654 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:36.655 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:36.655 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:19:36.655 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:19:36.655 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:19:36.655 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:36.656 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.652+0000 7f667a9d4640 10 monclient: shutdown 2024-03-25T18:19:36.656 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.652+0000 7f667a9d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:36.656 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.652+0000 7f667a9d4640 1 -- 172.21.15.156:0/3867574946 >> v1:172.21.15.194:6789/0 conn(0x7f66740b4d60 legacy=0x7f667413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:36.656 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.652+0000 7f667a9d4640 1 -- 172.21.15.156:0/3867574946 shutdown_connections 2024-03-25T18:19:36.656 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.652+0000 7f667a9d4640 1 -- 172.21.15.156:0/3867574946 >> 172.21.15.156:0/3867574946 conn(0x7f66740617a0 msgr2=0x7f6674061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:36.656 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.652+0000 7f667a9d4640 1 -- 172.21.15.156:0/3867574946 shutdown_connections 2024-03-25T18:19:36.657 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.652+0000 7f667a9d4640 1 -- 172.21.15.156:0/3867574946 wait complete. 2024-03-25T18:19:36.657 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.653+0000 7f667a9d4640 10 monclient: build_initial_monmap 2024-03-25T18:19:36.657 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.653+0000 7f667a9d4640 10 monclient: monmap: 2024-03-25T18:19:36.657 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:19:36.657 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:36.657 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:19:36.657 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:19:36.657 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:36.657 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:19:36.657 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:19:36.657 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:19:36.657 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:19:36.657 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:36.657 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.653+0000 7f667a9d4640 1 librados: starting msgr at 2024-03-25T18:19:36.658 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.653+0000 7f667a9d4640 1 librados: starting objecter 2024-03-25T18:19:36.658 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.654+0000 7f667a9d4640 1 Processor -- start 2024-03-25T18:19:36.658 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.654+0000 7f667a9d4640 1 -- start start 2024-03-25T18:19:36.658 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.654+0000 7f667a9d4640 1 librados: setting wanted keys 2024-03-25T18:19:36.658 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.654+0000 7f667a9d4640 1 librados: calling monclient init 2024-03-25T18:19:36.658 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.654+0000 7f667a9d4640 10 monclient: init 2024-03-25T18:19:36.659 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.655+0000 7f667a9d4640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:36.659 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.655+0000 7f667a9d4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:36.659 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.655+0000 7f667a9d4640 10 monclient(hunting): picked mon.noname-a con 0x7f66740b1730 addr v1:172.21.15.156:6789/0 2024-03-25T18:19:36.659 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.655+0000 7f667a9d4640 10 monclient(hunting): picked mon.noname-c con 0x7f66740b4d60 addr v1:172.21.15.156:6790/0 2024-03-25T18:19:36.659 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.655+0000 7f667a9d4640 10 monclient(hunting): picked mon.noname-b con 0x7f667414be90 addr v1:172.21.15.194:6789/0 2024-03-25T18:19:36.659 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.655+0000 7f667a9d4640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66741d78e0 con 0x7f66740b1730 2024-03-25T18:19:36.659 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.655+0000 7f667a9d4640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66741d7bc0 con 0x7f667414be90 2024-03-25T18:19:36.659 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.655+0000 7f667a9d4640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66741d8760 con 0x7f66740b4d60 2024-03-25T18:19:36.659 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.655+0000 7f667a9d4640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:36.660 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.655+0000 7f667a9d4640 10 monclient(hunting): authenticate will time out at 7657.967086s 2024-03-25T18:19:36.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.675+0000 7f6678f71640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f667414be90 0x7f6674137ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.156:49114/0 (socket says 172.21.15.156:49114) 2024-03-25T18:19:36.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.675+0000 7f6678f71640 1 -- 172.21.15.156:0/576247450 learned_addr learned my addr 172.21.15.156:0/576247450 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:19:36.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.675+0000 7f6670ff9640 10 client.?.objecter ms_handle_connect 0x7f667414be90 2024-03-25T18:19:36.677 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.675+0000 7f6670ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:36.679 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.677+0000 7f6670ff9640 10 client.?.objecter ms_handle_connect 0x7f66740b4d60 2024-03-25T18:19:36.680 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.677+0000 7f6670ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:36.682 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.680+0000 7f6670ff9640 10 client.?.objecter ms_handle_connect 0x7f66740b1730 2024-03-25T18:19:36.682 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.680+0000 7f6670ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:36.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.693+0000 7f6670ff9640 1 -- 172.21.15.156:0/576247450 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3170281157 0 0) 0x7f66741d7bc0 con 0x7f667414be90 2024-03-25T18:19:36.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.693+0000 7f6670ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:36.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.693+0000 7f6670ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:36.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.693+0000 7f6670ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:36.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.693+0000 7f6670ff9640 10 monclient(hunting): my global_id is 4741 2024-03-25T18:19:36.695 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.693+0000 7f6670ff9640 1 -- 172.21.15.156:0/576247450 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6660004eb0 con 0x7f667414be90 2024-03-25T18:19:36.698 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.696+0000 7f6670ff9640 1 -- 172.21.15.156:0/576247450 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1996009700 0 0) 0x7f66741d8760 con 0x7f66740b4d60 2024-03-25T18:19:36.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.696+0000 7f6670ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:36.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.696+0000 7f6670ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:36.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.696+0000 7f6670ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:36.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.696+0000 7f6670ff9640 10 monclient(hunting): my global_id is 4742 2024-03-25T18:19:36.699 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.696+0000 7f6670ff9640 1 -- 172.21.15.156:0/576247450 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66741d7bc0 con 0x7f66740b4d60 2024-03-25T18:19:36.702 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.700+0000 7f6670ff9640 1 -- 172.21.15.156:0/576247450 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660957635 0 0) 0x7f66741d78e0 con 0x7f66740b1730 2024-03-25T18:19:36.702 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.700+0000 7f6670ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:36.702 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.700+0000 7f6670ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:36.702 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.700+0000 7f6670ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:36.702 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.700+0000 7f6670ff9640 10 monclient(hunting): my global_id is 4734 2024-03-25T18:19:36.703 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.700+0000 7f6670ff9640 1 -- 172.21.15.156:0/576247450 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66741d8760 con 0x7f66740b1730 2024-03-25T18:19:36.705 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.702+0000 7f6670ff9640 1 -- 172.21.15.156:0/576247450 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 708716407 0 0) 0x7f6660004eb0 con 0x7f667414be90 2024-03-25T18:19:36.705 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.702+0000 7f6670ff9640 1 -- 172.21.15.156:0/576247450 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66741d78e0 con 0x7f667414be90 2024-03-25T18:19:36.705 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.702+0000 7f6670ff9640 1 -- 172.21.15.156:0/576247450 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f666c001c90 con 0x7f667414be90 2024-03-25T18:19:36.705 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.702+0000 7f6670ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:36.705 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.702+0000 7f6670ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:36.705 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.702+0000 7f6670ff9640 10 monclient(hunting): dump: 2024-03-25T18:19:36.705 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:19:36.705 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:36.705 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:36.706 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:36.706 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:36.706 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:36.706 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:19:36.706 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:19:36.706 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:19:36.706 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:36.706 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.702+0000 7f6670ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:36.718 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.715+0000 7f6670ff9640 1 -- 172.21.15.156:0/576247450 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3419709364 0 0) 0x7f66741d78e0 con 0x7f667414be90 2024-03-25T18:19:36.718 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.715+0000 7f6670ff9640 1 -- 172.21.15.156:0/576247450 >> v1:172.21.15.156:6790/0 conn(0x7f66740b4d60 legacy=0x7f66741d8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:36.718 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.715+0000 7f6670ff9640 1 -- 172.21.15.156:0/576247450 >> v1:172.21.15.156:6789/0 conn(0x7f66740b1730 legacy=0x7f6674142850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:36.718 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.715+0000 7f6670ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:19:36.718 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.715+0000 7f6670ff9640 1 monclient: found mon.b 2024-03-25T18:19:36.718 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.715+0000 7f6670ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:36.718 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.715+0000 7f6670ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:19:36.718 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.715+0000 7f6670ff9640 1 -- 172.21.15.156:0/576247450 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66740a9e20 con 0x7f667414be90 2024-03-25T18:19:36.718 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.715+0000 7f6670ff9640 10 monclient: _finish_auth 0 2024-03-25T18:19:36.718 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.715+0000 7f6670ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:19:36.719 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.715+0000 7f6670ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:36.719 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.715+0000 7f667a9d4640 5 monclient: authenticate success, global_id 4741 2024-03-25T18:19:36.719 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.715+0000 7f667a9d4640 10 monclient: _renew_subs 2024-03-25T18:19:36.719 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.715+0000 7f667a9d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:19:36.719 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.715+0000 7f667a9d4640 1 -- 172.21.15.156:0/576247450 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66740a96e0 con 0x7f667414be90 2024-03-25T18:19:36.719 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.715+0000 7f667a9d4640 10 client.4741.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:36.719 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.715+0000 7f667a9d4640 10 monclient: _renew_subs 2024-03-25T18:19:36.719 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.715+0000 7f667a9d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:19:36.719 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.715+0000 7f667a9d4640 1 -- 172.21.15.156:0/576247450 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6674140560 con 0x7f667414be90 2024-03-25T18:19:36.719 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.715+0000 7f667a9d4640 1 librados: init done 2024-03-25T18:19:36.720 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.717+0000 7f667a9d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:36.720 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.717+0000 7f667a9d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:36.720 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.717+0000 7f667a9d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:19:36.720 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.717+0000 7f667a9d4640 1 -- 172.21.15.156:0/576247450 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66740aa030 con 0x7f667414be90 2024-03-25T18:19:36.989 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.987+0000 7f6670ff9640 1 -- 172.21.15.156:0/576247450 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f666c003990 con 0x7f667414be90 2024-03-25T18:19:36.990 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.987+0000 7f6670ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:36.990 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.987+0000 7f6670ff9640 1 -- 172.21.15.156:0/576247450 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f666c003d60 con 0x7f667414be90 2024-03-25T18:19:36.990 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.987+0000 7f6670ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:36.990 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.987+0000 7f6670ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:36.990 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.987+0000 7f6670ff9640 10 monclient: dump: 2024-03-25T18:19:36.990 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:19:36.990 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:36.990 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:36.990 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:36.990 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:36.990 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:36.991 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:19:36.991 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:19:36.991 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:19:36.991 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:36.991 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.987+0000 7f66727fc640 4 set_mon_vals no callback set 2024-03-25T18:19:36.991 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.989+0000 7f6670ff9640 1 -- 172.21.15.156:0/576247450 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f666c003f80 con 0x7f667414be90 2024-03-25T18:19:36.992 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.989+0000 7f6673fff640 10 monclient: get_auth_request con 0x7f666006ddf0 auth_method 0 2024-03-25T18:19:36.992 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.989+0000 7f6670ff9640 10 client.4741.objecter ms_handle_connect 0x7f666006ddf0 2024-03-25T18:19:36.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.993+0000 7f6670ff9640 1 -- 172.21.15.156:0/576247450 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3381691338 0 0) 0x7f666c054a90 con 0x7f667414be90 2024-03-25T18:19:36.995 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.993+0000 7f6670ff9640 10 client.4741.objecter ms_dispatch 0x7f6674000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:19:36.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.993+0000 7f6670ff9640 3 client.4741.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:19:36.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.993+0000 7f6670ff9640 3 client.4741.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:19:36.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.993+0000 7f6670ff9640 20 client.4741.objecter dump_active .. 0 homeless 2024-03-25T18:19:36.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.993+0000 7f6670ff9640 1 -- 172.21.15.156:0/576247450 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f666c0505a0 con 0x7f667414be90 2024-03-25T18:19:36.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.993+0000 7f6670ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:36.996 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:36.993+0000 7f6670ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:37.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:37.243+0000 7f667a9d4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:37.245 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:37.243+0000 7f667a9d4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:37.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:37.243+0000 7f667a9d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:19:37.246 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:37.243+0000 7f667a9d4640 1 -- 172.21.15.156:0/576247450 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66741401f0 con 0x7f667414be90 2024-03-25T18:19:37.267 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:37.265+0000 7f6670ff9640 1 -- 172.21.15.156:0/576247450 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1441834839) 0x7f666c0524d0 con 0x7f667414be90 2024-03-25T18:19:37.268 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:37.265+0000 7f6670ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:37.268 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:37.265+0000 7f6670ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:37.268 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:19:37.268 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":11,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:18:20.029906+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:18:20.029906+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:37.271 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:37.267+0000 7f667a9d4640 10 librados: watch_flush enter 2024-03-25T18:19:37.271 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:37.268+0000 7f667a9d4640 10 librados: watch_flush exit 2024-03-25T18:19:37.271 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:37.268+0000 7f667a9d4640 20 client.4741.objecter shutdown clearing up homeless session... 2024-03-25T18:19:37.271 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:37.268+0000 7f667a9d4640 10 client.4741.objecter successfully canceled tick 2024-03-25T18:19:37.271 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:37.268+0000 7f667a9d4640 1 -- 172.21.15.156:0/576247450 >> v1:172.21.15.194:6800/2946427713 conn(0x7f666006ddf0 legacy=0x7f66600702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:37.271 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:37.268+0000 7f667a9d4640 10 monclient: shutdown 2024-03-25T18:19:37.271 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:37.268+0000 7f667a9d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:37.271 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:37.268+0000 7f667a9d4640 1 -- 172.21.15.156:0/576247450 >> v1:172.21.15.194:6789/0 conn(0x7f667414be90 legacy=0x7f6674137ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:37.271 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:37.268+0000 7f667a9d4640 1 -- 172.21.15.156:0/576247450 shutdown_connections 2024-03-25T18:19:37.271 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:37.268+0000 7f667a9d4640 1 -- 172.21.15.156:0/576247450 >> 172.21.15.156:0/576247450 conn(0x7f66740617a0 msgr2=0x7f6674061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:37.272 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:37.268+0000 7f667a9d4640 1 -- 172.21.15.156:0/576247450 shutdown_connections 2024-03-25T18:19:37.272 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:37.268+0000 7f667a9d4640 1 -- 172.21.15.156:0/576247450 wait complete. 2024-03-25T18:19:37.272 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:37.268+0000 7f667a9d4640 1 librados: shutdown 2024-03-25T18:19:37.282 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:44.089 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:44.205 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.203+0000 7f4807b15640 10 monclient: get_monmap_and_config 2024-03-25T18:19:44.205 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.203+0000 7f4807b15640 10 monclient: build_initial_monmap 2024-03-25T18:19:44.205 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.203+0000 7f4807b15640 10 monclient: monmap: 2024-03-25T18:19:44.205 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:19:44.205 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:44.205 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:19:44.204917+0000 2024-03-25T18:19:44.205 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:19:44.204917+0000 2024-03-25T18:19:44.205 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:44.205 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:44.205 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:19:44.205 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-25T18:19:44.205 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-b 2024-03-25T18:19:44.206 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:44.207 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.204+0000 7f4807b15640 1 Processor -- start 2024-03-25T18:19:44.207 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.204+0000 7f4807b15640 1 -- start start 2024-03-25T18:19:44.207 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.204+0000 7f4807b15640 10 monclient: init 2024-03-25T18:19:44.207 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.205+0000 7f4807b15640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:44.207 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.205+0000 7f4807b15640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:44.208 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.205+0000 7f4807b15640 10 monclient(hunting): picked mon.noname-b con 0x7f48000b4d60 addr v1:172.21.15.156:6790/0 2024-03-25T18:19:44.208 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.205+0000 7f4807b15640 10 monclient(hunting): picked mon.noname-a con 0x7f48000b1730 addr v1:172.21.15.156:6789/0 2024-03-25T18:19:44.208 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.205+0000 7f4807b15640 10 monclient(hunting): picked mon.noname-c con 0x7f480014be90 addr v1:172.21.15.194:6789/0 2024-03-25T18:19:44.208 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.205+0000 7f4807b15640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48000af940 con 0x7f48000b1730 2024-03-25T18:19:44.208 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.205+0000 7f4807b15640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48000b0a70 con 0x7f480014be90 2024-03-25T18:19:44.208 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.205+0000 7f4807b15640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48000b0d50 con 0x7f48000b4d60 2024-03-25T18:19:44.208 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.205+0000 7f4807b15640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:44.208 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.205+0000 7f4807b15640 10 monclient(hunting): authenticate will time out at 7665.516728s 2024-03-25T18:19:44.218 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.217+0000 7f48050b0640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f48000b1730 0x7f48000b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:40408/0 (socket says 172.21.15.156:40408) 2024-03-25T18:19:44.218 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.217+0000 7f48050b0640 1 -- 172.21.15.156:0/3059235394 learned_addr learned my addr 172.21.15.156:0/3059235394 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:19:44.224 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.223+0000 7f48048af640 1 -- 172.21.15.156:0/3059235394 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2795231615 0 0) 0x7f48000af940 con 0x7f48000b1730 2024-03-25T18:19:44.224 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.223+0000 7f48048af640 10 monclient(hunting): _negotiate 2024-03-25T18:19:44.224 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.223+0000 7f48048af640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:44.224 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.223+0000 7f48048af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:44.224 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.223+0000 7f48048af640 10 monclient(hunting): my global_id is 4737 2024-03-25T18:19:44.225 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.223+0000 7f48048af640 1 -- 172.21.15.156:0/3059235394 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47ec003520 con 0x7f48000b1730 2024-03-25T18:19:44.231 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.229+0000 7f48048af640 1 -- 172.21.15.156:0/3059235394 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301937494 0 0) 0x7f48000b0d50 con 0x7f48000b4d60 2024-03-25T18:19:44.231 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.229+0000 7f48048af640 10 monclient(hunting): _negotiate 2024-03-25T18:19:44.231 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.229+0000 7f48048af640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:44.231 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.229+0000 7f48048af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:44.231 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.229+0000 7f48048af640 10 monclient(hunting): my global_id is 4745 2024-03-25T18:19:44.231 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.229+0000 7f48048af640 1 -- 172.21.15.156:0/3059235394 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48000af940 con 0x7f48000b4d60 2024-03-25T18:19:44.231 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.229+0000 7f48048af640 1 -- 172.21.15.156:0/3059235394 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1094377747 0 0) 0x7f47ec003520 con 0x7f48000b1730 2024-03-25T18:19:44.231 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.229+0000 7f48048af640 1 -- 172.21.15.156:0/3059235394 >> v1:172.21.15.156:6790/0 conn(0x7f48000b4d60 legacy=0x7f480013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:44.231 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.229+0000 7f48048af640 1 -- 172.21.15.156:0/3059235394 >> v1:172.21.15.194:6789/0 conn(0x7f480014be90 legacy=0x7f480014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:44.231 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.229+0000 7f48048af640 10 monclient: _finish_hunting 0 2024-03-25T18:19:44.232 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.229+0000 7f48048af640 1 monclient: found mon.noname-a 2024-03-25T18:19:44.232 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.229+0000 7f48048af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:44.232 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.229+0000 7f48048af640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.156:6789/0 2024-03-25T18:19:44.232 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.229+0000 7f48048af640 1 -- 172.21.15.156:0/3059235394 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48000b1030 con 0x7f48000b1730 2024-03-25T18:19:44.232 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.229+0000 7f48048af640 10 monclient: _finish_auth 0 2024-03-25T18:19:44.232 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.229+0000 7f48048af640 10 monclient: _check_auth_tickets 2024-03-25T18:19:44.232 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.229+0000 7f48048af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:44.232 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.229+0000 7f48048af640 1 -- 172.21.15.156:0/3059235394 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f47f0001690 con 0x7f48000b1730 2024-03-25T18:19:44.232 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.229+0000 7f48048af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:44.232 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.229+0000 7f48048af640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:44.232 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.229+0000 7f48048af640 10 monclient: dump: 2024-03-25T18:19:44.232 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:19:44.232 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:44.232 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:44.232 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:44.232 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:44.233 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:44.233 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:19:44.233 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:19:44.233 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:19:44.233 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:44.233 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.229+0000 7f4807b15640 5 monclient: authenticate success, global_id 4737 2024-03-25T18:19:44.233 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.229+0000 7f4807b15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:44.234 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.233+0000 7f48048af640 1 -- 172.21.15.156:0/3059235394 <== mon.0 v1:172.21.15.156:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47f00017f0 con 0x7f48000b1730 2024-03-25T18:19:44.234 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.233+0000 7f48048af640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:44.235 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.233+0000 7f4807b15640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:44.235 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.233+0000 7f4807b15640 4 set_mon_vals no callback set 2024-03-25T18:19:44.236 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.234+0000 7f4807b15640 10 monclient: shutdown 2024-03-25T18:19:44.236 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.234+0000 7f4807b15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:44.236 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.234+0000 7f4807b15640 1 -- 172.21.15.156:0/3059235394 >> v1:172.21.15.156:6789/0 conn(0x7f48000b1730 legacy=0x7f48000b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:44.236 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.234+0000 7f4807b15640 1 -- 172.21.15.156:0/3059235394 shutdown_connections 2024-03-25T18:19:44.236 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.234+0000 7f4807b15640 1 -- 172.21.15.156:0/3059235394 >> 172.21.15.156:0/3059235394 conn(0x7f48000617a0 msgr2=0x7f4800061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:44.236 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.235+0000 7f4807b15640 1 -- 172.21.15.156:0/3059235394 shutdown_connections 2024-03-25T18:19:44.236 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.235+0000 7f4807b15640 1 -- 172.21.15.156:0/3059235394 wait complete. 2024-03-25T18:19:44.237 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.235+0000 7f4807b15640 10 monclient: build_initial_monmap 2024-03-25T18:19:44.237 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.235+0000 7f4807b15640 10 monclient: monmap: 2024-03-25T18:19:44.237 INFO:teuthology.misc.health.smithi156.stderr:epoch 0 2024-03-25T18:19:44.237 INFO:teuthology.misc.health.smithi156.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:44.237 INFO:teuthology.misc.health.smithi156.stderr:last_changed 0.000000 2024-03-25T18:19:44.237 INFO:teuthology.misc.health.smithi156.stderr:created 0.000000 2024-03-25T18:19:44.237 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:44.237 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 1 2024-03-25T18:19:44.237 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.noname-a 2024-03-25T18:19:44.237 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-25T18:19:44.237 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.noname-c 2024-03-25T18:19:44.238 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:44.238 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.236+0000 7f4807b15640 1 librados: starting msgr at 2024-03-25T18:19:44.238 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.236+0000 7f4807b15640 1 librados: starting objecter 2024-03-25T18:19:44.238 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.236+0000 7f4807b15640 1 Processor -- start 2024-03-25T18:19:44.238 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.236+0000 7f4807b15640 1 -- start start 2024-03-25T18:19:44.238 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.236+0000 7f4807b15640 1 librados: setting wanted keys 2024-03-25T18:19:44.238 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.236+0000 7f4807b15640 1 librados: calling monclient init 2024-03-25T18:19:44.238 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.236+0000 7f4807b15640 10 monclient: init 2024-03-25T18:19:44.239 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.236+0000 7f4807b15640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:44.239 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.236+0000 7f4807b15640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:44.239 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.236+0000 7f4807b15640 10 monclient(hunting): picked mon.noname-b con 0x7f48000b1730 addr v1:172.21.15.194:6789/0 2024-03-25T18:19:44.239 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.236+0000 7f4807b15640 10 monclient(hunting): picked mon.noname-c con 0x7f48000b4d60 addr v1:172.21.15.156:6790/0 2024-03-25T18:19:44.239 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.237+0000 7f4807b15640 10 monclient(hunting): picked mon.noname-a con 0x7f480014be90 addr v1:172.21.15.156:6789/0 2024-03-25T18:19:44.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.237+0000 7f4807b15640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4800139e50 con 0x7f480014be90 2024-03-25T18:19:44.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.237+0000 7f4807b15640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4800142800 con 0x7f48000b1730 2024-03-25T18:19:44.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.237+0000 7f4807b15640 1 -- --> v1:172.21.15.156:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4800140a90 con 0x7f48000b4d60 2024-03-25T18:19:44.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.237+0000 7f4807b15640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:44.240 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.237+0000 7f4807b15640 10 monclient(hunting): authenticate will time out at 7665.547884s 2024-03-25T18:19:44.248 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.247+0000 7f48060b2640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f480014be90 0x7f48001e7510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:40420/0 (socket says 172.21.15.156:40420) 2024-03-25T18:19:44.248 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.247+0000 7f48060b2640 1 -- 172.21.15.156:0/3144204397 learned_addr learned my addr 172.21.15.156:0/3144204397 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-03-25T18:19:44.248 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.247+0000 7f47f67fc640 10 client.?.objecter ms_handle_connect 0x7f480014be90 2024-03-25T18:19:44.249 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.247+0000 7f47f67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:44.259 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.257+0000 7f47f67fc640 10 client.?.objecter ms_handle_connect 0x7f48000b4d60 2024-03-25T18:19:44.259 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.257+0000 7f47f67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:44.259 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.258+0000 7f47f67fc640 10 client.?.objecter ms_handle_connect 0x7f48000b1730 2024-03-25T18:19:44.259 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.258+0000 7f47f67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:44.268 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.266+0000 7f47f67fc640 1 -- 172.21.15.156:0/3144204397 <== mon.0 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709111662 0 0) 0x7f4800139e50 con 0x7f480014be90 2024-03-25T18:19:44.268 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.266+0000 7f47f67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:44.268 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.266+0000 7f47f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:44.268 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.266+0000 7f47f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:44.268 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.266+0000 7f47f67fc640 10 monclient(hunting): my global_id is 4740 2024-03-25T18:19:44.268 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.266+0000 7f47f67fc640 1 -- 172.21.15.156:0/3144204397 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47e4004eb0 con 0x7f480014be90 2024-03-25T18:19:44.269 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.268+0000 7f47f67fc640 1 -- 172.21.15.156:0/3144204397 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060883467 0 0) 0x7f4800142800 con 0x7f48000b1730 2024-03-25T18:19:44.269 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.268+0000 7f47f67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:44.269 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.268+0000 7f47f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:44.269 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.268+0000 7f47f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:44.269 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.268+0000 7f47f67fc640 10 monclient(hunting): my global_id is 4747 2024-03-25T18:19:44.269 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.268+0000 7f47f67fc640 1 -- 172.21.15.156:0/3144204397 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4800139e50 con 0x7f48000b1730 2024-03-25T18:19:44.277 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.275+0000 7f47f67fc640 1 -- 172.21.15.156:0/3144204397 <== mon.2 v1:172.21.15.156:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899332779 0 0) 0x7f4800140a90 con 0x7f48000b4d60 2024-03-25T18:19:44.277 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.275+0000 7f47f67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:44.277 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.275+0000 7f47f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:44.277 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.275+0000 7f47f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:44.277 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.275+0000 7f47f67fc640 10 monclient(hunting): my global_id is 4748 2024-03-25T18:19:44.277 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.275+0000 7f47f67fc640 1 -- 172.21.15.156:0/3144204397 --> v1:172.21.15.156:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4800142800 con 0x7f48000b4d60 2024-03-25T18:19:44.277 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.275+0000 7f47f67fc640 1 -- 172.21.15.156:0/3144204397 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4042167584 0 0) 0x7f4800139e50 con 0x7f48000b1730 2024-03-25T18:19:44.277 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.275+0000 7f47f67fc640 1 -- 172.21.15.156:0/3144204397 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4800140a90 con 0x7f48000b1730 2024-03-25T18:19:44.277 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.275+0000 7f47f67fc640 1 -- 172.21.15.156:0/3144204397 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f47fc001cb0 con 0x7f48000b1730 2024-03-25T18:19:44.277 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.275+0000 7f47f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:44.277 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.275+0000 7f47f67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:44.277 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.275+0000 7f47f67fc640 10 monclient(hunting): dump: 2024-03-25T18:19:44.277 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:19:44.278 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:44.278 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:44.278 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:44.278 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:44.278 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:44.278 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:19:44.278 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:19:44.278 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:19:44.278 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:44.278 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.275+0000 7f47f67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:44.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.281+0000 7f47f67fc640 1 -- 172.21.15.156:0/3144204397 <== mon.0 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3776426315 0 0) 0x7f47e4004eb0 con 0x7f480014be90 2024-03-25T18:19:44.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.281+0000 7f47f67fc640 1 -- 172.21.15.156:0/3144204397 --> v1:172.21.15.156:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4800139e50 con 0x7f480014be90 2024-03-25T18:19:44.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.281+0000 7f47f67fc640 1 -- 172.21.15.156:0/3144204397 <== mon.0 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f47f8001910 con 0x7f480014be90 2024-03-25T18:19:44.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.281+0000 7f47f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:44.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.281+0000 7f47f67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:44.282 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.281+0000 7f47f67fc640 10 monclient(hunting): dump: 2024-03-25T18:19:44.282 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:19:44.283 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:44.283 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:44.283 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:44.283 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:44.283 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:44.283 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:19:44.283 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:19:44.283 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:19:44.283 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:44.295 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.293+0000 7f47f67fc640 1 -- 172.21.15.156:0/3144204397 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2558612043 0 0) 0x7f4800140a90 con 0x7f48000b1730 2024-03-25T18:19:44.295 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.293+0000 7f47f67fc640 1 -- 172.21.15.156:0/3144204397 >> v1:172.21.15.156:6790/0 conn(0x7f48000b4d60 legacy=0x7f4800134630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:44.295 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.293+0000 7f47f67fc640 1 -- 172.21.15.156:0/3144204397 >> v1:172.21.15.156:6789/0 conn(0x7f480014be90 legacy=0x7f48001e7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:44.295 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.293+0000 7f47f67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:44.295 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.293+0000 7f47f67fc640 1 monclient: found mon.b 2024-03-25T18:19:44.295 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.293+0000 7f47f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:44.295 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.293+0000 7f47f67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:19:44.295 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.293+0000 7f47f67fc640 1 -- 172.21.15.156:0/3144204397 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4800140c20 con 0x7f48000b1730 2024-03-25T18:19:44.295 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.293+0000 7f47f67fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:44.295 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.293+0000 7f47f67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:44.295 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.293+0000 7f47f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:44.295 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.293+0000 7f4807b15640 5 monclient: authenticate success, global_id 4747 2024-03-25T18:19:44.296 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.293+0000 7f4807b15640 10 monclient: _renew_subs 2024-03-25T18:19:44.296 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.293+0000 7f4807b15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:19:44.296 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.293+0000 7f4807b15640 1 -- 172.21.15.156:0/3144204397 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4800138ed0 con 0x7f48000b1730 2024-03-25T18:19:44.296 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.293+0000 7f4807b15640 10 client.4747.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:44.296 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.293+0000 7f4807b15640 10 monclient: _renew_subs 2024-03-25T18:19:44.296 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.293+0000 7f4807b15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:19:44.296 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.293+0000 7f4807b15640 1 -- 172.21.15.156:0/3144204397 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4800142f00 con 0x7f48000b1730 2024-03-25T18:19:44.296 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.293+0000 7f4807b15640 1 librados: init done 2024-03-25T18:19:44.297 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.294+0000 7f4807b15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:44.297 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.294+0000 7f4807b15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:44.297 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.294+0000 7f4807b15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:19:44.297 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.294+0000 7f4807b15640 1 -- 172.21.15.156:0/3144204397 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4800142990 con 0x7f48000b1730 2024-03-25T18:19:44.620 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.618+0000 7f47f67fc640 1 -- 172.21.15.156:0/3144204397 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47fc003990 con 0x7f48000b1730 2024-03-25T18:19:44.620 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.618+0000 7f47f67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:44.620 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.618+0000 7f47f67fc640 1 -- 172.21.15.156:0/3144204397 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4233022010 0 0) 0x7f47fc003d20 con 0x7f48000b1730 2024-03-25T18:19:44.620 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.618+0000 7f47f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:44.620 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.618+0000 7f47f67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:44.620 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.618+0000 7f47f67fc640 10 monclient: dump: 2024-03-25T18:19:44.620 INFO:teuthology.misc.health.smithi156.stderr:epoch 1 2024-03-25T18:19:44.620 INFO:teuthology.misc.health.smithi156.stderr:fsid af7d8066-e5a9-4d33-b2ab-862ad8bf139c 2024-03-25T18:19:44.620 INFO:teuthology.misc.health.smithi156.stderr:last_changed 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:44.620 INFO:teuthology.misc.health.smithi156.stderr:created 2024-03-25T18:07:23.148545+0000 2024-03-25T18:19:44.620 INFO:teuthology.misc.health.smithi156.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:44.620 INFO:teuthology.misc.health.smithi156.stderr:election_strategy: 3 2024-03-25T18:19:44.620 INFO:teuthology.misc.health.smithi156.stderr:0: v1:172.21.15.156:6789/0 mon.a 2024-03-25T18:19:44.621 INFO:teuthology.misc.health.smithi156.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-25T18:19:44.621 INFO:teuthology.misc.health.smithi156.stderr:2: v1:172.21.15.156:6790/0 mon.c 2024-03-25T18:19:44.621 INFO:teuthology.misc.health.smithi156.stderr: 2024-03-25T18:19:44.621 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.618+0000 7f47f7fff640 4 set_mon_vals no callback set 2024-03-25T18:19:44.621 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.619+0000 7f47f67fc640 1 -- 172.21.15.156:0/3144204397 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 381291792 0 0) 0x7f47fc003f40 con 0x7f48000b1730 2024-03-25T18:19:44.621 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.620+0000 7f48050b0640 10 monclient: get_auth_request con 0x7f47e406df70 auth_method 0 2024-03-25T18:19:44.622 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.620+0000 7f47f67fc640 10 client.4747.objecter ms_handle_connect 0x7f47e406df70 2024-03-25T18:19:44.625 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.624+0000 7f47f67fc640 1 -- 172.21.15.156:0/3144204397 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3381691338 0 0) 0x7f47fc0538c0 con 0x7f48000b1730 2024-03-25T18:19:44.625 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.624+0000 7f47f67fc640 10 client.4747.objecter ms_dispatch 0x7f4800000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:19:44.625 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.624+0000 7f47f67fc640 3 client.4747.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:19:44.625 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.624+0000 7f47f67fc640 3 client.4747.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:19:44.625 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.624+0000 7f47f67fc640 20 client.4747.objecter dump_active .. 0 homeless 2024-03-25T18:19:44.626 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.624+0000 7f47f67fc640 1 -- 172.21.15.156:0/3144204397 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f47fc04f3d0 con 0x7f48000b1730 2024-03-25T18:19:44.626 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.624+0000 7f47f67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:44.626 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.624+0000 7f47f67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:44.866 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.865+0000 7f4807b15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:44.866 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.865+0000 7f4807b15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:44.866 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.865+0000 7f4807b15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-25T18:19:44.866 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.865+0000 7f4807b15640 1 -- 172.21.15.156:0/3144204397 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48000627b0 con 0x7f48000b1730 2024-03-25T18:19:44.894 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.893+0000 7f47f67fc640 1 -- 172.21.15.156:0/3144204397 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1441834839) 0x7f47fc051300 con 0x7f48000b1730 2024-03-25T18:19:44.894 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.893+0000 7f47f67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:44.894 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.893+0000 7f47f67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:44.895 INFO:teuthology.misc.health.smithi156.stdout: 2024-03-25T18:19:44.895 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":11,"fsid":"af7d8066-e5a9-4d33-b2ab-862ad8bf139c","created":"2024-03-25T18:08:16.199694+0000","modified":"2024-03-25T18:18:20.029906+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:18:20.029906+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c34cf722-8f77-4343-a481-69e0c03a1ecd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eb5d1a35-4a00-4015-ad8e-a0d2d4a3d1fb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"069bc903-8f61-4312-a5fe-e5f02c2c695d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6345be70-da95-4347-b73b-80a6d24f36e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ad2db9-17b4-44d5-9d65-8d180c71733b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b7a3d42f-28d6-46b2-8ab1-33252d2b7ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a4b4b316-17b7-4a39-a61f-3004ce88b732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"40b4ef96-668b-4450-bf78-b59d1c4750b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:44.897 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.895+0000 7f4807b15640 10 librados: watch_flush enter 2024-03-25T18:19:44.897 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.895+0000 7f4807b15640 10 librados: watch_flush exit 2024-03-25T18:19:44.897 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.895+0000 7f4807b15640 20 client.4747.objecter shutdown clearing up homeless session... 2024-03-25T18:19:44.897 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.895+0000 7f4807b15640 10 client.4747.objecter successfully canceled tick 2024-03-25T18:19:44.897 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.895+0000 7f4807b15640 1 -- 172.21.15.156:0/3144204397 >> v1:172.21.15.194:6800/2946427713 conn(0x7f47e406df70 legacy=0x7f47e4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:44.897 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.895+0000 7f4807b15640 10 monclient: shutdown 2024-03-25T18:19:44.897 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.895+0000 7f4807b15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:44.897 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.895+0000 7f4807b15640 1 -- 172.21.15.156:0/3144204397 >> v1:172.21.15.194:6789/0 conn(0x7f48000b1730 legacy=0x7f4800140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:44.898 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.895+0000 7f4807b15640 1 -- 172.21.15.156:0/3144204397 shutdown_connections 2024-03-25T18:19:44.898 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.895+0000 7f4807b15640 1 -- 172.21.15.156:0/3144204397 >> 172.21.15.156:0/3144204397 conn(0x7f48000617a0 msgr2=0x7f480013c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:44.898 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.895+0000 7f4807b15640 1 -- 172.21.15.156:0/3144204397 shutdown_connections 2024-03-25T18:19:44.898 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.895+0000 7f4807b15640 1 -- 172.21.15.156:0/3144204397 wait complete. 2024-03-25T18:19:44.898 INFO:teuthology.misc.health.smithi156.stderr:2024-03-25T18:19:44.895+0000 7f4807b15640 1 librados: shutdown 2024-03-25T18:19:44.909 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:44.909 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T18:19:44.911 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T18:19:44.911 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T18:19:44.911 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:19:45.348 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:11:51.338 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:45.421 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:11:51.411 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:45.454 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:11:51.445 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:45.466 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:11:51.433 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:45.469 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:11:51.460 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:45.475 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:11:51.441 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:45.487 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:11:51.477 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:45.504 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:11:51.471 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:45.527 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:11:51.517 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:47.018 INFO:tasks.ceph.osd.0.smithi156.stderr:==00:00:11:25.209 35463== 2024-03-25T18:19:47.018 INFO:tasks.ceph.osd.0.smithi156.stderr:==00:00:11:25.209 35463== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:19:47.057 INFO:tasks.ceph.osd.0.smithi156.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:19:51.013 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:19:51.013 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi156 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T18:19:51.014 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T18:19:51.014 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T18:19:51.015 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:19:52.882 INFO:tasks.ceph.osd.1.smithi156.stderr:==00:00:11:31.086 35462== 2024-03-25T18:19:52.882 INFO:tasks.ceph.osd.1.smithi156.stderr:==00:00:11:31.086 35462== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:19:52.921 INFO:tasks.ceph.osd.1.smithi156.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:19:57.116 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:19:57.117 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi156 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T18:19:57.117 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T18:19:57.117 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T18:19:57.118 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:19:58.958 INFO:tasks.ceph.osd.2.smithi156.stderr:==00:00:11:37.148 35464== 2024-03-25T18:19:58.958 INFO:tasks.ceph.osd.2.smithi156.stderr:==00:00:11:37.148 35464== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:19:58.995 INFO:tasks.ceph.osd.2.smithi156.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:20:03.219 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:20:03.220 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi156 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T18:20:03.220 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T18:20:03.220 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T18:20:03.220 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:20:05.108 INFO:tasks.ceph.osd.3.smithi156.stderr:==00:00:11:43.288 35465== 2024-03-25T18:20:05.108 INFO:tasks.ceph.osd.3.smithi156.stderr:==00:00:11:43.288 35465== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:20:05.151 INFO:tasks.ceph.osd.3.smithi156.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:20:05.382 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:12:11.373 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:05.429 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:12:11.420 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:05.447 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:12:11.437 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:05.478 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:12:11.468 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:05.482 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:12:11.473 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:05.509 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:12:11.499 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:05.545 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:12:11.512 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:05.548 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:12:11.515 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:05.570 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:12:11.537 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:09.322 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:20:09.323 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi156 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T18:20:09.323 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T18:20:09.323 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T18:20:09.323 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:20:11.255 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:11:49.462 35554== 2024-03-25T18:20:11.255 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:11:49.462 35554== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:20:11.296 INFO:tasks.ceph.osd.4.smithi194.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:20:15.425 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:20:15.426 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi194 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T18:20:15.426 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T18:20:15.426 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T18:20:15.426 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:20:17.299 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:11:55.504 35555== 2024-03-25T18:20:17.299 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:11:55.504 35555== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:20:17.327 INFO:tasks.ceph.osd.5.smithi194.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:20:21.529 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:20:21.529 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi194 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T18:20:21.529 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T18:20:21.530 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T18:20:21.530 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:20:23.379 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:12:01.583 35556== 2024-03-25T18:20:23.379 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:12:01.583 35556== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:20:23.418 INFO:tasks.ceph.osd.6.smithi194.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:20:25.480 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:12:31.470 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:25.483 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:12:31.473 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:25.504 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:12:31.494 35149== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:25.527 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:12:31.494 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:25.531 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:12:31.498 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:25.536 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:12:31.526 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:25.540 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:12:31.531 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:25.545 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:12:31.512 35011== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:25.554 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:12:31.545 35148== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:27.632 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:20:27.632 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi194 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T18:20:27.633 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T18:20:27.633 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T18:20:27.633 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:20:29.197 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:12:07.397 35557== 2024-03-25T18:20:29.198 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:12:07.397 35557== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:20:29.228 INFO:tasks.ceph.osd.7.smithi194.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:20:33.735 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:20:33.736 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi194 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T18:20:33.736 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T18:20:33.736 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T18:20:33.737 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T18:20:33.737 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:20:33.765 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T18:20:33.765 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T18:20:33.765 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:20:33.766 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:20:33.765+0000 7f07ff151640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 35009) UID: 0 2024-03-25T18:20:33.766 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-25T18:20:33.765+0000 7f07ff151640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T18:20:33.793 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T18:20:33.793 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T18:20:33.793 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T18:20:33.793 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:20:33.821 INFO:tasks.ceph.mon.a.smithi156.stderr:2024-03-25T18:20:33.808+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35115) UID: 0 2024-03-25T18:20:33.821 INFO:tasks.ceph.mon.a.smithi156.stderr:2024-03-25T18:20:33.810+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:20:59.605 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:13:05.595 35149== 2024-03-25T18:20:59.605 INFO:tasks.ceph.mon.a.smithi156.stderr:==00:00:13:05.595 35149== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:20:59.615 INFO:tasks.ceph.mon.a.smithi156.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:21:04.299 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:21:04.300 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi156 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T18:21:04.300 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T18:21:04.301 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T18:21:04.301 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:21:04.318 INFO:tasks.ceph.mon.c.smithi156.stderr:2024-03-25T18:21:04.304+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35117) UID: 0 2024-03-25T18:21:04.319 INFO:tasks.ceph.mon.c.smithi156.stderr:2024-03-25T18:21:04.306+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T18:21:04.753 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:13:10.743 35148== 2024-03-25T18:21:04.754 INFO:tasks.ceph.mon.c.smithi156.stderr:==00:00:13:10.743 35148== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:21:04.763 INFO:tasks.ceph.mon.c.smithi156.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:21:10.402 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:21:10.402 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi156 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T18:21:10.402 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T18:21:10.403 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T18:21:10.403 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:21:10.420 INFO:tasks.ceph.mon.b.smithi194.stderr:2024-03-25T18:21:10.406+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 35007) UID: 0 2024-03-25T18:21:10.421 INFO:tasks.ceph.mon.b.smithi194.stderr:2024-03-25T18:21:10.408+0000 9f86640 -1 mon.b@1(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:21:10.941 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:13:16.907 35011== 2024-03-25T18:21:10.941 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:13:16.907 35011== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:21:10.951 INFO:tasks.ceph.mon.b.smithi194.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:21:16.504 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:21:16.504 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi194 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T18:21:16.504 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T18:21:16.504 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T18:21:16.505 DEBUG:teuthology.orchestra.run.smithi156:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T18:21:16.614 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi156.front.sepia.ceph.com 2024-03-25T18:21:16.615 DEBUG:teuthology.orchestra.run.smithi156:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T18:21:16.721 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi156.front.sepia.ceph.com 2024-03-25T18:21:16.721 DEBUG:teuthology.orchestra.run.smithi156:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T18:21:16.794 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi156.front.sepia.ceph.com 2024-03-25T18:21:16.794 DEBUG:teuthology.orchestra.run.smithi156:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T18:21:16.860 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi156.front.sepia.ceph.com 2024-03-25T18:21:16.861 DEBUG:teuthology.orchestra.run.smithi156:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T18:21:16.934 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi194.front.sepia.ceph.com 2024-03-25T18:21:16.934 DEBUG:teuthology.orchestra.run.smithi194:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T18:21:17.054 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi194.front.sepia.ceph.com 2024-03-25T18:21:17.054 DEBUG:teuthology.orchestra.run.smithi194:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T18:21:17.108 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi194.front.sepia.ceph.com 2024-03-25T18:21:17.108 DEBUG:teuthology.orchestra.run.smithi194:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T18:21:17.159 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi194.front.sepia.ceph.com 2024-03-25T18:21:17.159 DEBUG:teuthology.orchestra.run.smithi194:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T18:21:17.207 INFO:tasks.ceph:Archiving mon data... 2024-03-25T18:21:17.209 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621259/data/mon.a.tgz 2024-03-25T18:21:17.209 DEBUG:teuthology.orchestra.run.smithi156:> mktemp 2024-03-25T18:21:17.225 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.99O4iT3ACX 2024-03-25T18:21:17.226 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.99O4iT3ACX 2024-03-25T18:21:17.322 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.99O4iT3ACX 2024-03-25T18:21:17.407 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.99O4iT3ACX is 134KB 2024-03-25T18:21:17.459 DEBUG:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.99O4iT3ACX 2024-03-25T18:21:17.473 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621259/data/mon.c.tgz 2024-03-25T18:21:17.473 DEBUG:teuthology.orchestra.run.smithi156:> mktemp 2024-03-25T18:21:17.527 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.ORxY1wYM43 2024-03-25T18:21:17.527 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.ORxY1wYM43 2024-03-25T18:21:17.615 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.ORxY1wYM43 2024-03-25T18:21:17.699 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.ORxY1wYM43 is 136KB 2024-03-25T18:21:17.750 DEBUG:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.ORxY1wYM43 2024-03-25T18:21:17.765 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621259/data/mon.b.tgz 2024-03-25T18:21:17.765 DEBUG:teuthology.orchestra.run.smithi194:> mktemp 2024-03-25T18:21:17.782 INFO:teuthology.orchestra.run.smithi194.stdout:/tmp/tmp.8Sk0xRhXuv 2024-03-25T18:21:17.783 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.8Sk0xRhXuv 2024-03-25T18:21:17.888 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0666 /tmp/tmp.8Sk0xRhXuv 2024-03-25T18:21:17.975 DEBUG:teuthology.orchestra.remote:smithi194:/tmp/tmp.8Sk0xRhXuv is 136KB 2024-03-25T18:21:18.028 DEBUG:teuthology.orchestra.run.smithi194:> rm -fr /tmp/tmp.8Sk0xRhXuv 2024-03-25T18:21:18.043 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T18:21:18.043 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:21:18.045 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:21:18.427 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:21:18.696 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T18:21:18.698 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621259/remote/smithi156/crash 2024-03-25T18:21:18.699 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:21:18.726 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621259/remote/smithi194/crash 2024-03-25T18:21:18.727 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:21:18.754 INFO:tasks.ceph:Compressing logs... 2024-03-25T18:21:18.754 DEBUG:teuthology.orchestra.run.smithi156:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:21:18.770 DEBUG:teuthology.orchestra.run.smithi194:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:21:18.794 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29305.log 2024-03-25T18:21:18.794 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T18:21:18.840 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph.tmp-client.admin.29305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40405.log 2024-03-25T18:21:18.840 INFO:teuthology.orchestra.run.smithi156.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29305.log.gz 2024-03-25T18:21:18.840 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41342.log 2024-03-25T18:21:18.840 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.40405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39976.log 2024-03-25T18:21:18.840 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.41342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40522.log 2024-03-25T18:21:18.840 INFO:teuthology.orchestra.run.smithi156.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40405.log.gz 2024-03-25T18:21:18.840 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.39976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39742.log 2024-03-25T18:21:18.840 INFO:teuthology.orchestra.run.smithi156.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41342.log.gz 2024-03-25T18:21:18.840 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.40522.log: gzip -5 --verbose -- 86.5% /var/log/ceph/ceph-client.admin.35488.log -- replaced with /var/log/ceph/ceph-client.admin.39976.log.gz 2024-03-25T18:21:18.840 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-25T18:21:18.840 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.39742.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40522.log.gz 2024-03-25T18:21:18.840 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39310.log 2024-03-25T18:21:18.841 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.35488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40913.log 2024-03-25T18:21:18.841 INFO:teuthology.orchestra.run.smithi156.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39742.log.gz 2024-03-25T18:21:18.841 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.39310.log: 86.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.35488.log.gz 2024-03-25T18:21:18.841 INFO:teuthology.orchestra.run.smithi156.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38734.log 2024-03-25T18:21:18.841 INFO:teuthology.orchestra.run.smithi156.stderr: 86.8%/var/log/ceph/ceph-client.admin.40913.log: -- replaced with /var/log/ceph/ceph-client.admin.39310.log.gz 2024-03-25T18:21:18.841 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T18:21:18.841 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.38734.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T18:21:18.841 INFO:teuthology.orchestra.run.smithi156.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40913.log.gz 2024-03-25T18:21:18.841 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T18:21:18.841 INFO:teuthology.orchestra.run.smithi156.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38734.log.gz 2024-03-25T18:21:18.841 INFO:teuthology.orchestra.run.smithi156.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T18:21:18.841 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T18:21:18.841 INFO:teuthology.orchestra.run.smithi156.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T18:21:18.841 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T18:21:18.841 INFO:teuthology.orchestra.run.smithi156.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T18:21:18.841 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T18:21:18.842 INFO:teuthology.orchestra.run.smithi156.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T18:21:18.842 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35616.log 2024-03-25T18:21:18.842 INFO:teuthology.orchestra.run.smithi156.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T18:21:18.842 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41928.log 2024-03-25T18:21:18.842 INFO:teuthology.orchestra.run.smithi156.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T18:21:18.842 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.35616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41225.log 2024-03-25T18:21:18.842 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39115.log 2024-03-25T18:21:18.842 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.41928.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35616.log.gz 2024-03-25T18:21:18.842 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.41225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39703.log 2024-03-25T18:21:18.842 INFO:teuthology.orchestra.run.smithi156.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41928.log.gz 2024-03-25T18:21:18.842 INFO:teuthology.orchestra.run.smithi156.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41225.log.gz 2024-03-25T18:21:18.842 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39664.log 2024-03-25T18:21:18.842 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.39115.log: /var/log/ceph/ceph-client.admin.39703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39154.log 2024-03-25T18:21:18.842 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.39664.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39115.log.gz 2024-03-25T18:21:18.842 INFO:teuthology.orchestra.run.smithi156.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39703.log.gz 2024-03-25T18:21:18.843 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39469.log 2024-03-25T18:21:18.843 INFO:teuthology.orchestra.run.smithi156.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39664.log.gz 2024-03-25T18:21:18.843 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.39154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41460.log 2024-03-25T18:21:18.843 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.39469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40796.log 2024-03-25T18:21:18.843 INFO:teuthology.orchestra.run.smithi156.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39154.log.gz 2024-03-25T18:21:18.843 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.41460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39820.log 2024-03-25T18:21:18.843 INFO:teuthology.orchestra.run.smithi156.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39469.log.gz 2024-03-25T18:21:18.843 INFO:teuthology.orchestra.run.smithi156.stderr: 86.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.41460.log.gz 2024-03-25T18:21:18.843 INFO:teuthology.orchestra.run.smithi156.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41538.log 2024-03-25T18:21:18.843 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.40796.log: /var/log/ceph/ceph-client.admin.39820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35157.log 2024-03-25T18:21:18.843 INFO:teuthology.orchestra.run.smithi156.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40796.log.gz 2024-03-25T18:21:18.843 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.41538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41811.log 2024-03-25T18:21:18.843 INFO:teuthology.orchestra.run.smithi156.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39820.log.gz 2024-03-25T18:21:18.843 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.35157.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41538.log.gz 2024-03-25T18:21:18.843 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41694.log 2024-03-25T18:21:18.843 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.41811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40015.log 2024-03-25T18:21:18.843 INFO:teuthology.orchestra.run.smithi156.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.35157.log.gz 2024-03-25T18:21:18.844 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.41694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42201.log 2024-03-25T18:21:18.844 INFO:teuthology.orchestra.run.smithi156.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41811.log.gz 2024-03-25T18:21:18.844 INFO:teuthology.orchestra.run.smithi156.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41694.log.gz 2024-03-25T18:21:18.844 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42123.log 2024-03-25T18:21:18.844 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.40015.log: /var/log/ceph/ceph-client.admin.42201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40991.log 2024-03-25T18:21:18.844 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.42123.log: gzip 86.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.40015.log.gz --verbose 2024-03-25T18:21:18.844 INFO:teuthology.orchestra.run.smithi156.stderr: -- /var/log/ceph/ceph-client.admin.40874.log 2024-03-25T18:21:18.844 INFO:teuthology.orchestra.run.smithi156.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.42201.log.gz 2024-03-25T18:21:18.844 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.40991.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:21:18.844 INFO:teuthology.orchestra.run.smithi156.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.42123.log.gz 2024-03-25T18:21:18.844 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.40874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40601.log 2024-03-25T18:21:18.844 INFO:teuthology.orchestra.run.smithi156.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40991.log.gz 2024-03-25T18:21:18.844 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39859.log 2024-03-25T18:21:18.844 INFO:teuthology.orchestra.run.smithi156.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40874.log.gz 2024-03-25T18:21:18.844 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.40601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36451.log 2024-03-25T18:21:18.844 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.39859.log: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:21:18.845 INFO:teuthology.orchestra.run.smithi156.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40601.log.gz 2024-03-25T18:21:18.845 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42084.log 2024-03-25T18:21:18.845 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40952.log 2024-03-25T18:21:18.845 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.36451.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39859.log.gz 2024-03-25T18:21:18.845 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.42084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41577.log 2024-03-25T18:21:18.845 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.40952.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.36451.log.gz 2024-03-25T18:21:18.845 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40171.log 2024-03-25T18:21:18.845 INFO:teuthology.orchestra.run.smithi156.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42084.log.gz 2024-03-25T18:21:18.845 INFO:teuthology.orchestra.run.smithi156.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40952.log.gz 2024-03-25T18:21:18.845 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41186.log/var/log/ceph/ceph-client.admin.41577.log: 2024-03-25T18:21:18.845 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.40171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42162.log 2024-03-25T18:21:18.845 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.41186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41147.log 2024-03-25T18:21:18.845 INFO:teuthology.orchestra.run.smithi156.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41577.log.gz 2024-03-25T18:21:18.845 INFO:teuthology.orchestra.run.smithi156.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40171.log.gz 2024-03-25T18:21:18.846 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.42162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41030.log 2024-03-25T18:21:18.846 INFO:teuthology.orchestra.run.smithi156.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41186.log.gz 2024-03-25T18:21:18.846 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.41147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40483.log 2024-03-25T18:21:18.846 INFO:teuthology.orchestra.run.smithi156.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42162.log.gz 2024-03-25T18:21:18.846 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.41030.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41147.log.gz 2024-03-25T18:21:18.846 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41616.log 2024-03-25T18:21:18.846 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.40483.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41030.log.gz 2024-03-25T18:21:18.846 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T18:21:18.846 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.41616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39898.log 2024-03-25T18:21:18.846 INFO:teuthology.orchestra.run.smithi156.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40483.log.gz 2024-03-25T18:21:18.846 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-osd.1.log: gzip 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41616.log.gz -5 2024-03-25T18:21:18.846 INFO:teuthology.orchestra.run.smithi156.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42045.log 2024-03-25T18:21:18.846 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.39898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40444.log 2024-03-25T18:21:18.846 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.42045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41967.log 2024-03-25T18:21:18.846 INFO:teuthology.orchestra.run.smithi156.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39898.log.gz 2024-03-25T18:21:18.847 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.40444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40718.log 2024-03-25T18:21:18.847 INFO:teuthology.orchestra.run.smithi156.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42045.log.gz 2024-03-25T18:21:18.847 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.41967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40562.log 2024-03-25T18:21:18.847 INFO:teuthology.orchestra.run.smithi156.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40444.log.gz 2024-03-25T18:21:18.847 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.40718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41069.log 2024-03-25T18:21:18.847 INFO:teuthology.orchestra.run.smithi156.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41967.log.gz 2024-03-25T18:21:18.847 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.40562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40093.log 2024-03-25T18:21:18.847 INFO:teuthology.orchestra.run.smithi156.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40718.log.gz 2024-03-25T18:21:18.847 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.41069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39388.log 2024-03-25T18:21:18.847 INFO:teuthology.orchestra.run.smithi156.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40562.log.gz 2024-03-25T18:21:18.847 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.40093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40210.log 2024-03-25T18:21:18.847 INFO:teuthology.orchestra.run.smithi156.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41069.log.gz 2024-03-25T18:21:18.847 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.39388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40679.log 2024-03-25T18:21:18.847 INFO:teuthology.orchestra.run.smithi156.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40093.log.gz 2024-03-25T18:21:18.847 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.40210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39430.log 2024-03-25T18:21:18.847 INFO:teuthology.orchestra.run.smithi156.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39388.log.gz 2024-03-25T18:21:18.848 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.40679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41264.log 2024-03-25T18:21:18.848 INFO:teuthology.orchestra.run.smithi156.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40210.log.gz 2024-03-25T18:21:18.848 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.39430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41499.log 2024-03-25T18:21:18.848 INFO:teuthology.orchestra.run.smithi156.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40679.log.gz 2024-03-25T18:21:18.848 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37418.log 2024-03-25T18:21:18.848 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.41264.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39430.log.gz 2024-03-25T18:21:18.848 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.41499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39271.log 2024-03-25T18:21:18.848 INFO:teuthology.orchestra.run.smithi156.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41264.log.gz 2024-03-25T18:21:18.848 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.37418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39508.log 2024-03-25T18:21:18.848 INFO:teuthology.orchestra.run.smithi156.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41499.log.gz 2024-03-25T18:21:18.848 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41381.log 2024-03-25T18:21:18.848 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.39271.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37418.log.gz 2024-03-25T18:21:18.848 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.39508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41850.log 2024-03-25T18:21:18.848 INFO:teuthology.orchestra.run.smithi156.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39271.log.gz 2024-03-25T18:21:18.848 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.41381.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T18:21:18.848 INFO:teuthology.orchestra.run.smithi156.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39508.log.gz 2024-03-25T18:21:18.849 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41889.log 2024-03-25T18:21:18.849 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.41850.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41381.log.gz 2024-03-25T18:21:18.849 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- 86.2% /var/log/ceph/ceph-client.admin.42006.log 2024-03-25T18:21:18.849 INFO:teuthology.orchestra.run.smithi156.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41850.log.gz 2024-03-25T18:21:18.849 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.41889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39076.log 2024-03-25T18:21:18.849 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.42006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39625.log 2024-03-25T18:21:18.849 INFO:teuthology.orchestra.run.smithi156.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41889.log.gz 2024-03-25T18:21:18.849 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.39076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41108.log 2024-03-25T18:21:18.849 INFO:teuthology.orchestra.run.smithi156.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.42006.log.gz 2024-03-25T18:21:18.849 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.39625.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39076.log.gz 2024-03-25T18:21:18.849 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39232.log 2024-03-25T18:21:18.849 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39547.log 2024-03-25T18:21:18.849 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.41108.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39625.log.gz 2024-03-25T18:21:18.849 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.39232.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T18:21:18.849 INFO:teuthology.orchestra.run.smithi156.stderr: 86.6%/var/log/ceph/ceph-client.admin.39547.log: gzip -- replaced with /var/log/ceph/ceph-client.admin.41108.log.gz 2024-03-25T18:21:18.849 INFO:teuthology.orchestra.run.smithi156.stderr: 86.3% -5 --verbose -- /var/log/ceph/ceph-client.admin.39349.log 2024-03-25T18:21:18.850 INFO:teuthology.orchestra.run.smithi156.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39232.log.gz 2024-03-25T18:21:18.850 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42279.log 2024-03-25T18:21:18.850 INFO:teuthology.orchestra.run.smithi156.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39547.log.gz 2024-03-25T18:21:18.850 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.39349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39193.log 2024-03-25T18:21:18.850 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.42279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40249.log 2024-03-25T18:21:18.850 INFO:teuthology.orchestra.run.smithi156.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39349.log.gz 2024-03-25T18:21:18.850 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.39193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40757.log 2024-03-25T18:21:18.850 INFO:teuthology.orchestra.run.smithi156.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42279.log.gz/var/log/ceph/ceph-client.admin.40249.log: 2024-03-25T18:21:18.850 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35461.log 2024-03-25T18:21:18.850 INFO:teuthology.orchestra.run.smithi156.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39193.log.gz 2024-03-25T18:21:18.850 INFO:teuthology.orchestra.run.smithi156.stderr:gzip 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40249.log.gz 2024-03-25T18:21:18.850 INFO:teuthology.orchestra.run.smithi156.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T18:21:18.850 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.35461.log: /var/log/ceph/ceph-client.admin.40757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40835.log 2024-03-25T18:21:18.850 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-osd.0.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.35461.log.gz 2024-03-25T18:21:18.850 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39035.log 2024-03-25T18:21:18.851 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.40835.log: 86.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42240.log 2024-03-25T18:21:18.851 INFO:teuthology.orchestra.run.smithi156.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40757.log.gz 2024-03-25T18:21:18.851 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.39035.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40835.log.gz 2024-03-25T18:21:18.851 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41303.log 2024-03-25T18:21:18.851 INFO:teuthology.orchestra.run.smithi156.stderr: 87.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.39035.log.gz 2024-03-25T18:21:18.851 INFO:teuthology.orchestra.run.smithi156.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41733.log 2024-03-25T18:21:18.851 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.41303.log: /var/log/ceph/ceph-client.admin.42240.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:21:18.851 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.41733.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41303.log.gz 2024-03-25T18:21:18.851 INFO:teuthology.orchestra.run.smithi156.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42240.log.gz 2024-03-25T18:21:18.851 INFO:teuthology.orchestra.run.smithi156.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41733.log.gz 2024-03-25T18:21:18.851 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T18:21:18.851 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39781.log 2024-03-25T18:21:18.851 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40640.log 2024-03-25T18:21:18.851 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.39781.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39781.log.gz 2024-03-25T18:21:18.851 INFO:teuthology.orchestra.run.smithi156.stderr: 92.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:21:18.852 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41772.log 2024-03-25T18:21:18.852 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39586.log 2024-03-25T18:21:18.852 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.41772.log: /var/log/ceph/ceph-client.admin.40640.log: 86.0% 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40640.log.gz 2024-03-25T18:21:18.852 INFO:teuthology.orchestra.run.smithi156.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41772.log.gz 2024-03-25T18:21:18.852 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39937.log 2024-03-25T18:21:18.852 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38996.log 2024-03-25T18:21:18.852 INFO:teuthology.orchestra.run.smithi156.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T18:21:18.852 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.39937.log: /var/log/ceph/ceph-client.admin.39586.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39937.log.gz 2024-03-25T18:21:18.852 INFO:teuthology.orchestra.run.smithi156.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39586.log.gz 2024-03-25T18:21:18.852 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40132.log 2024-03-25T18:21:18.852 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.38996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41655.log 2024-03-25T18:21:18.852 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.40132.log: 86.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40327.log 2024-03-25T18:21:18.852 INFO:teuthology.orchestra.run.smithi156.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38996.log.gz 2024-03-25T18:21:18.852 INFO:teuthology.orchestra.run.smithi156.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40132.log.gz 2024-03-25T18:21:18.852 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.41655.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41655.log.gz 2024-03-25T18:21:18.853 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41421.log 2024-03-25T18:21:18.853 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.40327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40366.log 2024-03-25T18:21:18.853 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.41421.log: gzip 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40327.log.gz 2024-03-25T18:21:18.853 INFO:teuthology.orchestra.run.smithi156.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40288.log 2024-03-25T18:21:18.853 INFO:teuthology.orchestra.run.smithi156.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41421.log.gz 2024-03-25T18:21:18.853 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40054.log 2024-03-25T18:21:18.853 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.40366.log: /var/log/ceph/ceph-client.admin.40288.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T18:21:18.853 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.40054.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40288.log.gz 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40366.log.gz 2024-03-25T18:21:18.853 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-25T18:21:18.853 INFO:teuthology.orchestra.run.smithi156.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40054.log.gz 2024-03-25T18:21:18.853 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T18:21:18.854 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35455.log 2024-03-25T18:21:18.854 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35284.log 2024-03-25T18:21:18.854 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.35455.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T18:21:18.854 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.35284.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T18:21:18.854 INFO:teuthology.orchestra.run.smithi194.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35455.log.gz 2024-03-25T18:21:18.854 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T18:21:18.854 INFO:teuthology.orchestra.run.smithi194.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35284.log.gz 2024-03-25T18:21:18.854 INFO:teuthology.orchestra.run.smithi194.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T18:21:18.854 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T18:21:18.854 INFO:teuthology.orchestra.run.smithi194.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T18:21:18.854 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T18:21:18.854 INFO:teuthology.orchestra.run.smithi194.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T18:21:18.854 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35164.log 2024-03-25T18:21:18.854 INFO:teuthology.orchestra.run.smithi194.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T18:21:18.854 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35244.log 2024-03-25T18:21:18.855 INFO:teuthology.orchestra.run.smithi194.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T18:21:18.855 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.35164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35364.log 2024-03-25T18:21:18.855 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.35244.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:21:18.855 INFO:teuthology.orchestra.run.smithi194.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35164.log.gz 2024-03-25T18:21:18.855 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.35364.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T18:21:18.855 INFO:teuthology.orchestra.run.smithi194.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35244.log.gz 2024-03-25T18:21:18.855 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T18:21:18.855 INFO:teuthology.orchestra.run.smithi194.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35364.log.gz 2024-03-25T18:21:18.855 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T18:21:18.855 INFO:teuthology.orchestra.run.smithi194.stderr: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:21:18.855 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35324.log 2024-03-25T18:21:18.855 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35415.log 2024-03-25T18:21:18.855 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.35324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35204.log 2024-03-25T18:21:18.855 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.35415.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T18:21:18.855 INFO:teuthology.orchestra.run.smithi194.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35324.log.gz 2024-03-25T18:21:18.855 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.35204.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:21:18.856 INFO:teuthology.orchestra.run.smithi194.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35415.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T18:21:18.856 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:21:18.856 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-mgr.x.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35204.log.gz 2024-03-25T18:21:18.856 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: 86.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:21:18.911 INFO:teuthology.orchestra.run.smithi194.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T18:21:19.644 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-mon.c.log: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T18:21:19.650 INFO:teuthology.orchestra.run.smithi156.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T18:21:19.697 INFO:teuthology.orchestra.run.smithi194.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T18:21:19.731 INFO:teuthology.orchestra.run.smithi194.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T18:21:19.761 INFO:teuthology.orchestra.run.smithi194.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T18:21:19.942 INFO:teuthology.orchestra.run.smithi156.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T18:21:19.952 INFO:teuthology.orchestra.run.smithi156.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T18:21:20.040 INFO:teuthology.orchestra.run.smithi194.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T18:21:23.881 INFO:teuthology.orchestra.run.smithi156.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T18:21:24.089 INFO:teuthology.orchestra.run.smithi156.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T18:21:24.091 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-25T18:21:24.091 INFO:teuthology.orchestra.run.smithi156.stderr:real 0m5.309s 2024-03-25T18:21:24.091 INFO:teuthology.orchestra.run.smithi156.stderr:user 0m13.826s 2024-03-25T18:21:24.092 INFO:teuthology.orchestra.run.smithi156.stderr:sys 0m0.555s 2024-03-25T18:21:24.131 INFO:teuthology.orchestra.run.smithi194.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T18:21:24.133 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-25T18:21:24.133 INFO:teuthology.orchestra.run.smithi194.stderr:real 0m5.318s 2024-03-25T18:21:24.133 INFO:teuthology.orchestra.run.smithi194.stderr:user 0m8.964s 2024-03-25T18:21:24.133 INFO:teuthology.orchestra.run.smithi194.stderr:sys 0m0.350s 2024-03-25T18:21:24.133 INFO:tasks.ceph:Archiving logs... 2024-03-25T18:21:24.133 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621259/remote/smithi156/log 2024-03-25T18:21:24.135 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:21:26.366 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621259/remote/smithi194/log 2024-03-25T18:21:26.367 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:21:27.890 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:21:28.084 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f0fb3d97d45a49c78f826e77a5d1b35c Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:21:28.088 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T18:21:28.098 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T18:21:28.148 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T18:21:28.148 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:21:28.162 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:21:28.242 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T18:21:28.254 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T18:21:28.254 DEBUG:teuthology.orchestra.run.smithi156:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:21:28.257 DEBUG:teuthology.orchestra.run.smithi194:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:21:28.272 INFO:teuthology.orchestra.run.smithi194.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:21:28.272 INFO:teuthology.orchestra.run.smithi156.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:21:28.276 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:21:28.276 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-03-25T18:21:28.277 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 50 +578us[ +578us] +/- 95ms 2024-03-25T18:21:28.277 INFO:teuthology.orchestra.run.smithi194.stdout:^* hv02.front.sepia.ceph.com 3 6 377 55 -1295us[-1297us] +/- 77ms 2024-03-25T18:21:28.277 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 +740us[ +740us] +/- 66ms 2024-03-25T18:21:28.277 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:21:28.278 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:21:28.278 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2024-03-25T18:21:28.278 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 56 +594us[ +591us] +/- 95ms 2024-03-25T18:21:28.278 INFO:teuthology.orchestra.run.smithi156.stdout:^* hv02.front.sepia.ceph.com 3 6 377 48 -1283us[-1286us] +/- 77ms 2024-03-25T18:21:28.279 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 +760us[ +757us] +/- 66ms 2024-03-25T18:21:28.279 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:21:28.279 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T18:21:28.290 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T18:21:28.290 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T18:21:28.343 DEBUG:teuthology.orchestra.run.smithi156:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:21:28.416 DEBUG:teuthology.orchestra.run.smithi194:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:21:28.482 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:21:28.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:21:28.514 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:21:28.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:21:28.543 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T18:21:28.555 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T18:21:28.569 INFO:teuthology.task.internal:Duration was 7429.818910 seconds 2024-03-25T18:21:28.569 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T18:21:28.584 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T18:21:28.584 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:21:28.586 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:21:28.629 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:21:28.629 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:21:28.923 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T18:21:28.923 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi156.front.sepia.ceph.com 2024-03-25T18:21:28.924 DEBUG:teuthology.orchestra.run.smithi156:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:21:28.943 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi194.front.sepia.ceph.com 2024-03-25T18:21:28.943 DEBUG:teuthology.orchestra.run.smithi194:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:21:28.986 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T18:21:28.987 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:21:28.989 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:21:29.055 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T18:21:29.056 DEBUG:teuthology.orchestra.run.smithi156:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:21:29.058 DEBUG:teuthology.orchestra.run.smithi194:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:21:29.200 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T18:21:29.209 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T18:21:29.210 DEBUG:teuthology.orchestra.run.smithi156:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:21:29.237 DEBUG:teuthology.orchestra.run.smithi194:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:21:29.264 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T18:21:29.273 DEBUG:teuthology.orchestra.run.smithi156:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:21:29.279 DEBUG:teuthology.orchestra.run.smithi194:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:21:29.302 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = core 2024-03-25T18:21:29.329 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = core 2024-03-25T18:21:29.350 DEBUG:teuthology.orchestra.run.smithi156:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:21:29.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:21:29.379 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:21:29.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:21:29.404 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T18:21:29.415 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T18:21:29.416 DEBUG:teuthology.misc:Transferring archived files from smithi156:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621259/remote/smithi156 2024-03-25T18:21:29.416 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:21:29.456 DEBUG:teuthology.misc:Transferring archived files from smithi194:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621259/remote/smithi194 2024-03-25T18:21:29.456 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:21:29.491 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T18:21:29.491 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:21:29.493 DEBUG:teuthology.orchestra.run.smithi194:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:21:29.540 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T18:21:29.554 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T18:21:29.554 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T18:21:29.570 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T18:21:29.570 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:21:29.573 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:21:29.585 INFO:teuthology.orchestra.run.smithi156.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:21 /home/ubuntu/cephtest 2024-03-25T18:21:29.594 INFO:teuthology.orchestra.run.smithi194.stdout: 265945 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:21 /home/ubuntu/cephtest 2024-03-25T18:21:29.596 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T18:21:29.609 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T18:21:29.669 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T18:21:29.695 DEBUG:teuthology.nuke:shortname: smithi156 2024-03-25T18:21:29.695 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:21:29.718 DEBUG:teuthology.nuke:shortname: smithi194 2024-03-25T18:21:29.718 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:21:29.721 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621259', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:08:58.009581', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCkGg4kPQJMrq/462q94BLEniTaliNpRqetozH5/BjX8Z+YsVbRb9yqGlYNox0HbEpukRiW9mhXvMGQTIUutda4='} 2024-03-25T18:21:29.740 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621259', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:08:58.012069', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL8bN/RvARiv4DZo2Ft70wIhEmZyDAQhciYVLOvdDAaTOzX9xpc3r1GJHnSvl0cmcRUaESnAh3PhVN2JQjiPfsE='} 2024-03-25T18:21:29.768 INFO:teuthology.orchestra.console.smithi156:Power off 2024-03-25T18:21:29.769 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:21:29.783 INFO:teuthology.orchestra.console.smithi194:Power off 2024-03-25T18:21:29.784 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:21:29.793 DEBUG:teuthology.orchestra.console.smithi156:power off output: Chassis Power Control: Down/Off 2024-03-25T18:21:29.793 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:21:29.806 DEBUG:teuthology.orchestra.console.smithi194:power off output: Chassis Power Control: Down/Off 2024-03-25T18:21:29.806 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:21:29.816 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is on 2024-03-25T18:21:29.828 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-03-25T18:21:33.816 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:21:33.829 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:21:33.927 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is on 2024-03-25T18:21:33.939 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-03-25T18:21:37.932 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:21:37.939 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:21:38.041 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off 2024-03-25T18:21:38.049 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-03-25T18:21:38.142 INFO:teuthology.orchestra.console.smithi156:Power off completed 2024-03-25T18:21:42.053 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:21:42.164 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is off 2024-03-25T18:21:42.264 INFO:teuthology.orchestra.console.smithi194:Power off completed 2024-03-25T18:21:42.369 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 7429.818910360336 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=f0fb3d97d45a49c78f826e77a5d1b35c status: fail success: false 2024-03-25T18:21:42.370 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:21:42.622 INFO:teuthology.run:FAIL